Solving Cloudflare TurnstileA Technical Postmortem

At 02:14 a realtime price feed stops producing rows. Nothing pages anyone. The transport layer reports no outage, requests are completing, and the affected source is answering 200 OK. By every signal the HTTP client understands, the pipeline is healthy. By the only signal the business cares about, the feed has gone dark.

The problem is in the body. The source has started serving a Cloudflare Turnstile interstitial instead of prices, the client accepted the 200 as success, and the challenge page flowed downstream to be parsed as if it were data. There was a transport-level success signal and no content-level signal at all.

This post is written as the postmortem you would file afterwards. The incident is a representative one, reconstructed from how this failure usually presents, and the fix is deliberately not a Turnstile solver. It adds challenge detection at the response boundary, classifies every response as ok, challenge, or hard_block, and routes challenged fetches through the Crawlbase JavaScript token, which renders the page and handles the challenge upstream. The pipeline's job shrinks to noticing the condition, choosing a transport, and verifying what comes back.

A representative timeline

• T+0 (02:14). Price rows stop arriving and dashboards flatten.
• T+9m. On-call starts looking. Logs show 200 OK for the affected source, so the transport looks fine and attention goes elsewhere.
• T+18m. Someone captures a raw response body. The document is titled Just a moment... and carries the Turnstile widget.
• T+24m. Root cause: the ingestion path defines success as HTTP status. In this failure mode the challenge arrives as 200, though 403 is just as common.
• T+41m. A staging fix detects the challenge and reroutes the affected URL through the JavaScript token.
• T+58m. Rows resume. The captured body is kept as a fixture, so detection can be tested without waiting for the source to challenge again.

Cloudflare introducing a challenge is not the finding; sources change protection all the time. The finding is that the pipeline had no way to tell a successful HTTP response from successful content, which collapsed four operationally different situations into one boolean.

The shape of the fix

Four pieces, each small on purpose:

• A detector: a pure function of { status, headers, html } that reports which challenge signals are present. No network calls, which is what makes saved responses replayable.
• A classifier that turns those signals into ok, challenge, or hard_block. A generic "failed" state cannot tell the pipeline what to do next.
• Two transports with the same response shape: directFetch, the path that failed, and crawlbaseFetch, the remediation.
• An incident log that records detection, classification, and remediation as a timeline you can paste into the postmortem.

The detector's purity is the part that pays off longest. Once detection is a function over saved input, the exact bytes from the outage become a test that runs on every change, and "did we fix it" stops depending on whether the source happens to be challenging right now.

The runnable code lives in ScraperHub/solving-cloudflare-turnstile-a-technical-postmortem, with the finished version under final/ and staged checkpoints under steps/. The snippets below are excerpts from final/.

Environment

Node.js 18 or newer, for the built-in fetch, and a Crawlbase account. The remediation route uses the JavaScript token. That follows our own escalation rule for the Crawling API: a Normal-token request that comes back empty or with 525, meaning the challenge could not be solved, should be retried on the JavaScript token, and Turnstile interstitials are the situation that rule exists for. The detection and classification steps need no token at all.

git clone https://github.com/ScraperHub/solving-cloudflare-turnstile-a-technical-postmortem.git
cd solving-cloudflare-turnstile-a-technical-postmortem/final
npm install
cp .env.example .env  # then set CRAWLBASE_JS_TOKEN

Step 1: configuration

One module reads the environment. The token is optional at load time and only required on the remediation path, so an incident can be reproduced and triaged before anyone has fetched credentials.

const config = {
  crawlbaseJsToken: process.env.CRAWLBASE_JS_TOKEN || '',
  controlUrl: process.env.CONTROL_URL || 'https://example.com',
  targetUrl: process.env.TARGET_URL || 'https://crawlbase.com/blog',
  requestTimeoutMs: Number(process.env.REQUEST_TIMEOUT_MS || 20000),
};

The control URL is the quiet hero. It is a page that should never trip the detector, so if it does, the regression is in your detection logic rather than the target. Without it, a detector that flags everything looks identical to a source that challenges everything.

Step 2: a detector you can replay

The original logic treated status as proof of content. The detector replaces that assumption by looking at the body and headers, and returns what it found rather than a verdict.

const HTML_MARKERS = [
  'challenges.cloudflare.com/turnstile',
  'cf-turnstile',
  '__cf_chl_',
  'cf_chl_opt',
  'window._cf_chl_opt',
  'Just a moment',
  'Checking your browser',
];

const CHALLENGE_HEADERS = ['cf-mitigated'];

function detect({ status, headers = {}, html = '' }) {
  const lowerHeaders = {};
  for (const [key, value] of Object.entries(headers)) {
    lowerHeaders[key.toLowerCase()] = String(value).toLowerCase();
  }

  const hitMarkers = HTML_MARKERS.filter((marker) =>
    html.toLowerCase().includes(marker.toLowerCase())
  );

  const cfMitigated =
    CHALLENGE_HEADERS.some((h) => lowerHeaders[h]) &&
    (lowerHeaders['cf-mitigated'] || '').includes('challenge');

  const servedByCloudflare =
    (lowerHeaders['server'] || '').includes('cloudflare') ||
    Boolean(lowerHeaders['cf-ray']);

  const hasTurnstileWidget = hitMarkers.some(
    (m) => m === 'cf-turnstile' || m === 'challenges.cloudflare.com/turnstile'
  );

  return {
    status,
    servedByCloudflare,
    cfMitigated,
    hasTurnstileWidget,
    challengeMarkers: hitMarkers,
    challengeDetected: cfMitigated || hitMarkers.length > 0,
  };
}

The markers are the strings a Cloudflare challenge actually carries: the Turnstile script URL, the cf-turnstile container, the __cf_chl_ and cf_chl_opt challenge namespaces, the interstitial title, and the cf-mitigated: challenge header. The detector checks for their presence and nothing more. It never touches the widget.

Run it against the body captured during the outage:

npm run detect -- fixtures/turnstile-challenge.html

It should report outcome: "challenge" along with the markers it matched. That fixture is the most valuable file in the repository: challenge pages change, and the saved response turns any future detector edit into something you can prove did not quietly stop recognising the thing that took the feed down.

One honest limit of substring matching belongs here. 'Just a moment' and 'cf-turnstile' are plain text, so any page that merely mentions them matches too. Point TARGET_URL at this article and the detector will classify it as a challenge, because the article quotes every marker in the list. That is exactly the kind of false positive the control URL and the known-good fixture exist to catch, and in production it argues for requiring a structural signal, such as the header or the widget script, before trusting a text-only match.

Step 3: three outcomes, not two

The detector says what is present. The classifier turns that into something the pipeline can act on.

const OUTCOME = {
  OK: 'ok',
  CHALLENGE: 'challenge',
  HARD_BLOCK: 'hard_block',
};

function classify(signals) {
  if (signals.challengeDetected) {
    return OUTCOME.CHALLENGE;
  }
  if (signals.status === 200) {
    return OUTCOME.OK;
  }
  if ([403, 429, 503].includes(signals.status) && signals.servedByCloudflare) {
    return OUTCOME.HARD_BLOCK;
  }
  return signals.status === 200 ? OUTCOME.OK : OUTCOME.HARD_BLOCK;
}

Order is the whole design. The challenge check runs first because an interstitial arrives as either 200 or 403. Let the 200 check win and the classifier reproduces the incident by construction.

Read the tail carefully, too. Once the first two branches have returned, the last two can only ever produce hard_block: the Cloudflare-specific branch and the fallback agree. So every response that is neither a challenge nor a 200 is a hard block, including a 404, a 500, and a request that timed out and came back with status 0. That is a conservative default for a demo and the first thing to split in production, because a transient 503 deserves a retry and a hard block deserves a back-off.

• ok: no challenge signals and a 200. Pass it on to normal content validation.
• challenge: a detectable Cloudflare challenge. Reroute through the JavaScript token.
• hard_block: no usable content and no challenge to hand off. Back off the host or change strategy; retrying the same request the same way will not help.

Step 4: remediate through Crawlbase

The remediation is a second transport returning the same shape as directFetch, so detection, classification, and logging never need to know which path produced a response. Error handling is trimmed from this excerpt.

async function crawlbaseFetch(url, token) {
  const started = Date.now();
  const endpoint = `https://api.crawlbase.com/?token=${token}&url=${encodeURIComponent(url)}`;
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), config.requestTimeoutMs);
  try {
    const response = await fetch(endpoint, { signal: controller.signal });
    const html = await response.text();
    return {
      transport: 'crawlbase-js',
      status: Number(response.headers.get('cb_status') || response.status),
      originalStatus: Number(response.headers.get('original_status') || 0),
      headers: normalizeHeaders(response.headers),
      html,
      latencyMs: Date.now() - started,
    };
  } finally {
    clearTimeout(timer);
  }
}

Two headers carry the verdict. cb_status is Crawlbase's result for the fetch; original_status is what the target answered. Branch on cb_status, and treat 525 as its own case: it means the challenge could not be solved, which calls for a retry and then a back-off, not a store.

The rerouted response goes back through the same detector and classifier before anything is stored. Changing transport is not evidence that a challenge was handled. The content is accepted only when cb_status reports success and the body is free of challenge signals.

Here is how the companion repository wires the two transports together:

if (directResult.outcome !== OUTCOME.OK) {
  if (!config.crawlbaseJsToken) {
    incident.warn('CRAWLBASE_JS_TOKEN not set; cannot run the remediation path.');
  } else {
    incident.fix('Routing target through Crawlbase JavaScript token.');
    const viaCrawlbase = await crawlbaseFetch(config.targetUrl, config.crawlbaseJsToken);
    const crawlbaseResult = inspect(viaCrawlbase);
    incident.log(
      crawlbaseResult.outcome === OUTCOME.OK ? 'fix' : 'warn',
      `target via crawlbase-js -> ${crawlbaseResult.outcome} ` +
        `(cb_status ${viaCrawlbase.status}, ${viaCrawlbase.latencyMs}ms)`
    );
  }
}

Note the condition: !== OUTCOME.OK. The demo reroutes anything that is not clean, hard blocks included, which is the simplest thing that recovers the feed. It is also the line to change first. A hard block has no challenge to hand off, so sending it through the remediation path spends money and latency on a request that will likely fail the same way. Once the outcomes exist, give each one its own exit:

// Production shape, not from the repository: one exit per outcome.
switch (directResult.outcome) {
  case OUTCOME.OK:
    return store(direct);
  case OUTCOME.CHALLENGE:
    return rerouteAndVerify(url);
  case OUTCOME.HARD_BLOCK:
    return backOff(new URL(url).host);
}

Running it

npm start

The tool replays the saved fixtures, checks the control URL, then fetches the target directly, logging each step as part of the incident timeline:

# Turnstile challenge on realtime ingestion

T+0.0s  [WARN] Replaying saved fixtures from the outage window.
T+0.0s  [WARN] fixture turnstile-challenge.html -> challenge (markers: ...)
T+0.0s  [OK] fixture ok-page.html -> ok (no false positive expected)
T+0.1s  [OK] control https://example.com -> ok (http 200, 133ms)
T+1.0s  [OK] target https://crawlbase.com/blog via direct -> ok (http 200, 843ms)

Notice what that run proves and what it does not. The live target answered cleanly, so no reroute happened and no Crawlbase request was spent. The challenge path was exercised by the fixture replay instead, which is the point of keeping the fixture: detection is verified deterministically, without needing the source to challenge at the exact moment you run the tool. When the target does serve a challenge, the direct result classifies as challenge, the URL goes out on the JavaScript token, and the returned body is inspected again before it counts.

Production considerations

Gate writes on the outcome, never on the status. This is the single control that would have prevented the incident. A 200 is permission to inspect, not permission to store.

Keep the fixture, and add to it. Challenge pages change their markup. Every new variant you capture becomes another regression case, and the known-good fixture keeps the other half honest by proving the detector is not flagging normal pages.

Split hard blocks from transient errors. The demo's fallback folds timeouts, 5xx responses, and genuine blocks into one outcome. Give transient failures a bounded retry and reserve back-off for real blocks, or a brief upstream wobble will pause a healthy host.

Don't reroute what you can't fix. Only challenge goes through the JavaScript token. Rerouting hard blocks inflates reroute volume and cost without recovering content.

Use the cheapest token that works. Most traffic belongs on the direct path or the Normal token. Escalating per response, rather than defaulting everything to the JavaScript token, keeps latency and spend proportional to how much of your traffic is actually challenged.

Watch the reroute rate as a signal. A sudden rise in challenge outcomes for one host is the early warning the original pipeline never had. Alert on it, and you find out about the next protection change before the dashboards flatten.

Keep the scope explicit. Fetch only sources you are authorized to access, and respect their terms and rate expectations. The example uses example.com as the control and the Crawlbase blog as the target for exactly that reason.

For the wider picture of how these protections work, inside modern anti-bot evasion covers the systems view, and avoiding Cloudflare bot detection covers the request-level side.

Conclusion

The outage was never really about Cloudflare. It was about a pipeline that let HTTP status stand in for content, so the moment a source answered with a challenge page, the failure became invisible.

The fix makes it visible and then makes it a decision. A pure detector that can be replayed against the exact bytes from the outage. A classifier with three outcomes that checks for a challenge before it trusts a 200. A second transport that sends challenged fetches through the JavaScript token, and a second inspection that refuses to store anything until the body is clean. The application never solves anything; it notices, routes, and verifies.

To reproduce it, create a free Crawlbase account and clone ScraperHub/solving-cloudflare-turnstile-a-technical-postmortem.

Frequently Asked Questions (FAQs)

Does this bypass Cloudflare Turnstile?

The application does not. It detects the challenge and hands the affected request to the Crawling API on the JavaScript token, which renders the page and handles the challenge upstream as part of the managed fetch. There is no challenge-solving logic in the pipeline, and the same access rules apply as for any fetch: only sources you are authorized to access.

Why did the client get 200 OK for a challenge?

Because the interstitial is itself a valid HTTP response. Cloudflare can serve a challenge page with 200 or 403. A pipeline that defines success by status code will store the challenge page as data, which is exactly what happened here.

Should I use the Normal token or the JavaScript token?

Start with the cheapest token that works and escalate on evidence. Our documented rule is that a Normal-token request returning an empty body or 525 should be retried on the JavaScript token. Turnstile interstitials are the textbook case for that escalation, which is why the remediation route in this post goes straight to the JavaScript token.

How do I know the reroute actually worked?

Two conditions, both required: cb_status reports success, and the returned body passes the detector with no challenge signals. A 525 means the challenge could not be solved; retry, and if it persists back off and investigate, since that usually means the target has rolled out a new challenge variant.

Why keep hard_block separate from challenge?

Because they call for opposite actions. A challenge has something to hand off, so rerouting can recover the content. A hard block does not, so rerouting just repeats a failure at higher cost. Collapsing them into one "failed" state is how pipelines end up either retrying blocks forever or never recovering from challenges.

NZ

Neil Zamora

Senior Architect · Crawlbase

Senior architect at Crawlbase, focused on the systems behind large-scale crawling: proxy rotation, anti-bot resilience, and the APIs that hide that complexity.