Bot Detection in 2026Diagnosing Scraper Failures Across Cloudflare, DataDome and Akamai

A scraper that worked a month ago starts returning 403s, challenge pages, or HTTP 200 responses with the data missing. The first suspect is usually a fresh anti-bot release from Cloudflare, DataDome or Akamai, and sometimes that is right. Just as often the cause is a changed page template, an expired session cookie, a new redirect, or a worker pool that quietly doubled its concurrency.

Before rotating proxies or rebuilding browser fingerprints, find out where the failure actually happens. That matters more in 2026 than it used to, because detection increasingly looks past the single request: session behavior, browser integrity, and how a client moves through a site over time can all decide whether the next request is allowed or challenged.

This post covers what the three vendors actually documented in 2026, what each change means for a scraping pipeline, and a practical way to diagnose failures, escalate selectively and measure recovery. The code lives in ScraperHub/what-changed-in-cloudflare-datadome-and-akamai-bot-detection-in-2026, a diagnostic framework in Python that classifies responses and reports usable outcomes. It does not solve challenges or change request identity, and nothing here does either.

What changed in bot detection in 2026

None of the relevant developments is a single new fingerprint or one mandatory challenge. They change when a decision is made, what evidence informs it, and how site operators can treat different types of automated traffic.

These are documented announcements and releases, not evidence that every protected site runs them. Several capabilities were announced without a public general-availability date.

Cloudflare: detection continues past the first request

Precursor, introduced in July, evaluates browser and behavioral signals over a session. Cloudflare's documentation explains that the evolving assessment updates session state stored in the cf_clearance cookie, which can be reduced or invalidated. For a scraper, passing the first challenge is no longer a stable result: on zones where Precursor is enabled, later requests can change the verdict.

Availability needs checking per zone. The launch changelog talks about rolling out to all customers, while the announcement frames Precursor as part of Enterprise Bot Management, free until general availability later in the year.

Adaptive Intelligence followed on August 31. Its model is retrained continuously on live traffic and is enabled through Auto Update Machine Learning for Enterprise Bot Management customers, which makes periodic numbered model releases less of a landmark. The announcement described disposable, short-lived rules as a component still to come, so treat those as planned rather than shipped.

In parallel, Cloudflare split AI crawler policy into separate Search, Agent and Training controls. From September 15, new zones get recommended presets, stricter for ad-funded sites, which operators can change at any time. These are operator choices, not a ban on all automated requests, and Bot Preference Sync can write the chosen preferences into robots.txt.

DataDome: challenge integrity and browser evidence

DataDome's VM-based obfuscation compiles the Device Check and Slider logic into bytecode that a custom in-browser interpreter runs. The main effect is to make the challenge harder to reverse-engineer and replay cheaply, not to add a new category of network fingerprint.

Its Proof of Browser write-up describes checks that combine rendering, WebGL, DOM mutations and timing. These are vendor-described mechanisms; neither this post nor the companion repository benchmarks how well they detect anything. The Device Check documentation explains how detection results trigger the challenge.

DataDome also packaged Agent Trust on May 5, after adding Web Bot Auth support in January. Identity evidence can come from signed authentication or other trusted mechanisms. Copying a known crawler's user agent establishes neither identity nor permission.

Finally, the 3.0 releases of DataDome's Apache, Nginx, Cloudflare Worker and Fastly modules changed path handling and static-asset exclusion. They are integration changes that depend on each operator upgrading, not a newly launched TLS or HTTP/2 detector.

Akamai: adaptive validation and finer bot policy

In March, Akamai announced Content Protector enhancements, including native-app protection and the Advanced Validation Framework. AVF can request more device or behavioral data when passive analysis is inconclusive, with an opt-in interactive challenge. The announcement gives no public general-availability date.

Akamai also split its AI-bot traffic into more granular categories: Training Crawlers, Search Crawlers, and Fetchers and Agents, with matching filters in Web Security Analytics. They give operators finer control and do not prescribe a default action.

The sources reviewed here give no public basis for claiming that Akamai shipped a new core Bot Manager model, a new _abck format or a new residential-proxy detector in 2026. The same restraint applies to familiar techniques elsewhere: TLS fingerprinting, browser fingerprints and IP reputation still matter, but their existence is not a 2026 announcement.

Diagnose the failure before changing your scraper

Start from one preserved failing response. Record its timestamp, requested and final URL, response headers and body, scraper version, exit location, session identifier where relevant, and concurrency, and compare it with a known-good capture. Without those dimensions, linking an incident to a vendor release is mostly guesswork.

Rule out the common application failures first:

1. Markup or parser drift: does the expected element still exist in the raw HTML?
2. JavaScript rendering: does the data only appear after client-side execution or in a later network response?
3. Authentication: did the request redirect to a login, lose a cookie or hit an expired CSRF token?
4. Traffic pattern: did the worker count, request rate or exit location change?
5. Redirects and variants: did the canonical URL, locale, device version or content type change?

Then classify the response itself, not the exception your parser raised.

Vendor markers narrow the search but are not reliable attribution on their own. Cloudflare documents cf-mitigated: challenge; a cf-ray header alone is not a block verdict, because every proxied response carries one. DataDome cookies or challenge endpoints add context, and Akamai-associated cookie names such as _abck and bm_sz prove nothing about a specific enforcement decision.

Keep the fetch verdict apart from content success

The Crawling API makes this distinction observable. cb_status is Crawlbase's verdict on the response, including its own block and content checks. original_status is normally the status the target sent. The final url exposes redirects, and with format=json these fields arrive alongside the body and the domain_complexity pricing tier. A request to https://example.com/ on October 7, 2026 returned:

{
  "original_status": 200,
  "pc_status": 200,
  "cb_status": 200,
  "url": "https://example.com/",
  "domain_complexity": "standard"
}

None of these statuses says that a product title, price or any other business field is valid. A robust pipeline validates those fields explicitly, as a third layer.

The companion repository's classifier, in final/bot_diag/classify.py, makes the distinction concrete:

# final/bot_diag/classify.py
def classify_response(
    status: int | None,
    body: str,
    headers: Mapping[str, str] | None = None,
    expected_fields: Sequence[str] | None = None,
) -> str:
    header_map = _header_map(headers)
    text = body or ""
    lowered = text.lower()
    if _is_cloudflare_challenge(lowered, header_map) or _is_datadome_challenge(lowered, header_map):
        return LABEL_CHALLENGED
    is_blocked = (
        _is_akamai_block(text, lowered)
        or _is_cloudflare_block(lowered, header_map)
        or _is_datadome_block(status, lowered, header_map)
        or (status in {401, 403, 429} and bool(text.strip()))
    )
    if is_blocked:
        return LABEL_BLOCKED
    return _content_label(status, text, expected_fields)

This is an excerpt, not a standalone snippet: the imports, labels and helpers live elsewhere in the package. It returns one of four labels, blocked, challenged, invalid_content and ok. The field validator is deliberately small and supports simple JSON and HTML patterns rather than production schemas, so read step 3 and its fixtures before adapting it to your data.

A practical extension is to validate field types, plausible ranges, cross-field consistency and freshness, not just presence. That catches the failure described in silent response poisoning, where a technically successful fetch still returns unusable or misleading values.

Choose recovery by failure class

Different failures need different interventions. Blind rotation and unconditional retries make the evidence harder to read and add load and cost.

Request and session consistency. A claimed browser identity has to be coherent with its transport, headers, cookie history and navigation sequence. Swapping only the user agent or the exit IP rarely helps against browser-integrity or session-level assessment. Keep exits stable where a workflow depends on logged-in or stateful navigation.

Browser execution. Render when the fields you need depend on JavaScript. It costs more and is slower: on the Crawling API a JavaScript request is billed at twice the standard rate. Rendering is not permission to access a target, and it is not a universal challenge solution. For the operational trade-off, see headless browsers versus scraping APIs.

Retry and escalation. Back off on transient failures with a cap. Fix invalid URLs and selectors instead of retrying them unchanged. Investigate a confirmed block rather than hammering it with the same request. On the Crawling API, these are the codes you will meet most often and what to do first:

A failed call also comes back with HTTP status 520. That is the status line, not a cb_status, so always branch on the cb_status in the response. The repository's escalation step keys its JavaScript escalation on 520 and 525 and adds page_wait=2000 and ajax_wait. Map it to the table above before reusing it: escalate on 503 and on an unexpected 207, and treat 705 and 706 as a signal to slow down. The waits help with late-rendering content; they are not a remedy for a failed challenge.

Identity and policy. For named, legitimate agents, signed identity and registration can be a better route than imitating arbitrary browser traffic. Cloudflare Web Bot Auth, DataDome Agent Trust and Akamai's agent-related controls belong to this layer. Verified identity still does not guarantee authorization.

Managed retrieval. Crawlbase offers the Crawling API for request-oriented integrations, Smart AI Proxy for existing proxy-compatible clients, and the Enterprise Crawler for queued bulk jobs and delivery. Managed infrastructure removes the need to run browser fleets and proxy pools yourself, but supported targets, latency, credits and failure rates still need measuring.

Evaluate recovery with the companion project

The repository needs Python 3.10+ and provides six incremental scripts under steps/ plus a bot_diag package under final/. It is a diagnostic framework, not a live bypass benchmark for any vendor. Its pipeline:

1. Start with a Normal-token Crawling API request using format=json.
2. Inspect cb_status, original_status, the final URL and the content.
3. Classify the response and validate the required fields.
4. Retry or escalate only where the failure policy warrants it.
5. Save one result per URL to CSV and summarize usable outcomes and retry overhead.
6. Show a per-domain choice between Smart AI Proxy and the Crawling API.

Run a reproducible baseline

Clone the repository, install its dependencies, and start with the synthetic path, which needs no token. From the final directory:

python -m bot_diag evaluate urls.example.txt --fields price,title --dry-run

For a live run, set CRAWLBASE_TOKEN (and CRAWLBASE_JS_TOKEN for JavaScript escalation), provide a URL list you are authorized to fetch, and run:

python -m bot_diag evaluate urls.txt --fields price,title \
  --concurrency 4 --max-retries 2 --out sample_results.csv

The evaluator caps concurrency at eight. It reports an ok rate over input URLs, nearest-rank p50 and p95 latency per request attempt, and retry overhead. The latency figure excludes retry backoff sleeps, and the CSV holds one consolidated row per URL, not one per attempt. Both matter when you compare the numbers with a production job's wall-clock time.

For a meaningful comparison, sample roughly 100 to 200 URLs across representative page types, apply the same validation rules to every fetch method, and repeat on more than one day. Segment the results by domain, page class, geography and rendering mode. Synthetic fixtures test program flow; only live, authorized samples measure target reliability.

Route a failing domain deliberately

The routing example counts observed challenges per normalized hostname. Below its default threshold of three it keeps Smart AI Proxy; at or above it, it moves the domain to the Crawling API:

# final/bot_diag/routing.py
def route_domain(
    domain: str,
    challenge_counts: Mapping[str, int],
    *,
    threshold: int = DEFAULT_CHALLENGE_THRESHOLD,
) -> str:
    """Return ``api`` after ``threshold`` challenges, otherwise ``proxy``.

    ``proxy`` means the caller's existing HTTP client should keep using
    ``smartproxy.crawlbase.com:8013``. ``api`` means that domain should move
    to the Crawling API. Counts must already be keyed by ``domain_key``.
    """
    if threshold < 1:
        raise ValueError("threshold must be at least 1")
    key = domain_key(domain)
    if challenge_counts.get(key, 0) >= threshold:
        return API
    return PROXY

Step 6 prints the decision; it does not open a proxy connection or fail over automatically. Smart AI Proxy's HTTPS endpoint is smartproxy.crawlbase.com:8013 with token-based proxy authentication, and the Smart AI Proxy documentation covers its CrawlbaseAPI-Parameters header, including JavaScript options. In the example, Crawling API escalation uses the separate JavaScript token.

A real routing service also needs expiring counters, observability, circuit-breaker behavior and a path back to the cheaper route. Without them, a short burst of three challenges moves a hostname's traffic for good.

Measure cost per usable result, not per HTTP 200

Once the same validated URL sample has run through two approaches, compare reliability and total operating cost together.

url_completion_rate    = urls_with_valid_final_result / input_urls
attempt_success_rate   = successful_attempts / all_fetch_attempts
cost_per_usable_result = total_allocated_cost / validated_results

The denominators differ on purpose. URL completion measures how much of the job finished; attempt success measures how efficient the fetch path is. Do not put the two side by side and call both a "success rate".

Count proxy and compute spend, managed-provider credits, retry overhead, browser infrastructure and engineering maintenance, without double-counting what a managed rate already includes. On Crawlbase, only successful calls are charged: target 200s, 404s, 410s and redirects, empty 200 bodies (cb_status 207), and any status you opt into with custom_success_codes. Blocks and internal failures are free. domain_complexity is the pricing tier that multiplies each charged request, so read actual consumption from your account usage and Crawlbase pricing rather than estimating it from request counts.

Self-managed clients can be economical on predictable targets or when full browser control is essential. Managed retrieval becomes more attractive once maintaining domain-specific workarounds eats engineering time. Neither removes the need to validate the data that comes back.

Conclusion

The significant 2026 changes are about continuous evidence, browser integrity and finer operator policy. None of them retires the old failure modes: broken selectors, expired sessions, uncontrolled concurrency and incomplete rendering.

Treat scraping reliability as an observable system: keep raw responses, separate the fetch verdict from the target's answer, classify failures conservatively, escalate selectively, and judge the result by validated data. The companion repository is a starting point for that workflow. If your fetch infrastructure is becoming the maintenance bottleneck, run the same representative URLs through Crawlbase and compare the cost per usable result before changing the architecture.

Primary references

• Cloudflare: Precursor, Precursor documentation, Adaptive Intelligence, AI traffic controls, mixed-use AI crawler policies
• DataDome: VM-based obfuscation, Proof of Browser, Agent Trust, Device Check
• Akamai: Content Protector enhancements, AI bot traffic categories, Web Security Analytics update

Frequently Asked Questions (FAQs)

Why does a scraper pass a challenge and fail later?

On sites using session-based assessment, later activity can change the trust decision. Compare request order, cookies, session continuity, rate and exit behavior before assuming the first challenge was the only enforcement point.

Does an HTTP 200 mean the scrape succeeded?

No. It confirms a response at one layer. A challenge page, an empty shell, a redirected page or plausible but wrong data can all arrive with a 200. Validate the fields and what they mean before marking a URL complete.

Do residential proxies still help against modern bot detection?

They help with IP reputation and geographic access, not with missing browser execution or inconsistent session signals. Aggressive per-request rotation can make stateful workflows less coherent, not more.

Which Crawling API responses are charged?

Only successful ones: cb_status 200 (including target 404, 410 and redirects), empty 200 bodies reported as cb_status 207, and statuses you opt into with custom_success_codes. Blocks such as 503, 704, 705 and 706 and internal failures are not charged.

Can Crawlbase guarantee access to Cloudflare, DataDome or Akamai protected sites?

No provider can guarantee every target and configuration. The right test is a repeatable, authorized URL sample with field-level validation, latency and credit accounting, not a single successful request.

NZ

Neil Zamora

Senior Architect · Crawlbase

Senior architect at Crawlbase, focused on the systems behind large-scale crawling: proxy rotation, anti-bot resilience, and the APIs that hide that complexity.

IK

Ian Kalvin

Technical Support Engineer · Crawlbase

Technical support engineer at Crawlbase, writing from the front line of what actually breaks in production scraping and proxy setups.