The Web Has Started Grading CrawlersTrust Bands, 429s as a Contract, and What a Good Crawler Looks Like Now

For most of the web's history, a crawler was judged once, at the door. A server looked at the user agent, perhaps the IP, and decided to let the request in or not. What the crawler did afterwards was nobody's business until a site fell over, and even then the complaint went to an abuse mailbox that nobody read.

In the third week of October 2026 that changed, three times in four days. On 13 October, Google documented that its Google-Agent fetcher signs a subset of its requests with Web Bot Auth. On 15 and 16 October, a research crawler identifying itself with a Roblox address read the robots.txt of three hobbyist 3D model sites, ignored it, and pulled 145.5 GB. And on 16 October, Cloudflare announced that Verified bots are now scored continuously on whether they honour robots.txt and whether they slow down after a 429.

Put together, those three events describe the new rule. Identity gets you in the door. Conduct decides whether you are still welcome next month. For anyone who runs a crawler, the question is no longer "can I fetch this?" but "what will the record say I did?"

Three things that happened in one week

The crawler first. Open3DLab, SFMLab and SmutBase are three small sites run by the same people, hosting user-made 3D models. On 26 October they published their logs: over 34 hours on 15 and 16 October, a client calling itself trellis2-data-sourcing/1.0 (3D foundation model research; [redacted]@roblox.com) made 26,429 requests from a single AWS address, read the pages of 18,926 projects and downloaded 1,027 files from 712 of them. It fetched robots.txt and did not follow it. 423 of the projects were licensed for non-commercial use only. When the sites blocked it on 18 October, it came back on the 19th and again on the 20th, and, in the sites' words, "did not reduce its download attempts when faced with our blocks". Roblox, according to a statement reported later, said a researcher's agent had scraped the sites by mistake and had been decommissioned.

Nothing about this crawler was hidden. It had a user agent with an email address in it. It came from one IP. It was, by the standards of the Verified bots programme, honest about its identity. It was also the exact behaviour that programme now measures and penalises, and the sites could prove it from their own logs, which is why the story went to the front page of Hacker News rather than to an abuse mailbox.

Cloudflare's announcement, published the day after the crawler's main run, is short on mechanism and precise on intent. "Continuous accountability for Verified bots" describes two initial signals. The first asks whether a bot's observed traffic is consistent with the site's published preferences over time, with RFC 9309 as the reference for how robots.txt is read. The second asks what happens after a site returns a 429: "Does it slow down, pause, or does it continue sending requests at the same pace?" The results feed an "explainable trust band", a label the post is careful to call a history rather than a verdict, because "isolated violations" count for little and "repeated patterns of behavior that disregards site preferences" count for a lot. Scores will be published on Radar, and site owners will be able to choose the band at which Verified traffic is still allowed through.

Google's contribution is quieter and in some ways more consequential. Its Web Bot Auth page, updated 13 October, says that a subset of requests from Google-Agent, the fetcher behind agents hosted on Google infrastructure, are signed and authenticate as https://agent.bot.goog. The implementation is marked experimental. But the Google-Agent page sits in the list of user-triggered fetchers, and the guidance for that list has not changed: "Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules." So the largest crawler operator on earth is now signing requests from an agent that, by its own documentation, does not read the file everyone else is being scored on. That is not hypocrisy; it is the shape of the argument to come. Signatures settle who sent a request. They say nothing about whether the request was welcome.

What a trust band actually measures

It helps to be concrete about scale. At Cloudflare Connect, held from 19 to 21 October, Matthew Conroy, the company's senior director of product management, said that agents made more than 600 billion requests on its network last quarter, almost four times the figure at the end of 2025, and that "fewer than one in ten automated requests comes from a bot that identifies itself". Matthew Prince's forecast in the same session was that agent traffic would reach a thousand times human traffic within five years, with the caveat that his earlier predictions had all been too conservative. DataDome's Q3 report, published 15 October, adds the detail that explains why identity is suddenly a product: in a sample of traffic from Meta's Muse agent, 99.3% of requests came from cloud and CDN ranges, 81.2% of them Cloudflare's own, and the agent does not declare itself. IP reputation, the tool the whole industry leaned on for twenty years, cannot distinguish an agent from the edge network it runs on.

So the scoring has to rest on conduct, and conduct has to be observed over time. That is what the trust band is. Read as an engineering spec rather than a policy post, it has four inputs.

Identity is the precondition, not the score. Web Bot Auth, the Ed25519 signature scheme we covered when it shipped alongside pay-per-crawl, has been an IETF working-group document since September and was revised on 29 October as draft-ietf-webbotauth-httpsig-protocol-01. OpenAI and AWS sign; Google is experimenting; Cloudflare's own Web Search API, launched 2 October with Ceramic.ai, Exa and Linkup behind it, notes that all three "have committed to Cloudflare's verified bot crawling standards". Without a stable identity there is nothing to attach a history to, which is why unsigned traffic is not scored at all. It is simply the default-deny class.

Published preferences are the first signal, and the vocabulary is still moving. RFC 9309 is the baseline: groups, Allow and Disallow, longest-match wins. On top of it sit Cloudflare's Content Signals, which carry purpose rather than path: search, ai-input, ai-train, and since 31 August a fourth field, use, with the values reference and full. Cloudflare's own crawl endpoint now rejects a request with a 400 when the crawler's declared contentUse is more permissive than the site's. The IETF's parallel effort, the AI Preferences vocabulary, reached draft 08 on 14 September and is still an Internet-Draft; the working group's August milestone passed without a standard. The scoring, in other words, started before the grammar was finished.

Response to feedback is the second signal and the one most crawlers fail without noticing. A 429 is not an error to retry; it is an instruction, and when it carries Retry-After it is an instruction with a number in it. Cloudflare's post puts it plainly: slowing down or pausing after a 429 "increases trust, while repeatedly ignoring the signal lowers it". Most retry libraries do the opposite. They retry the failed request with a short delay and leave the concurrency that caused the 429 exactly where it was.

Efficiency is not in the initial scoring and will be. Cloudflare's July press release claimed that "over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages", and its pilot with OpenAI, running since July on about 1% of its network, feeds the crawler change signals and filters for pages that do not exist or merely redirect. The post reports hundreds of millions of newly discovered pages, and one sentence worth keeping: "A signal about a page doesn't grant permission to access it." The signals are being built for crawlers that already behave; they are not a way in.

Behind all four sits money. Pay Per Use, in beta since 30 September, pays a publisher when an AI company reports that it used a page, not when it fetched it, because, as Cloudflare puts it, "charging for every crawl makes the buyer pay before it knows what it needs". Monetization Gateway, the same day, answers a request with 402 Payment Required and a price, settled over the x402 protocol. Both need a buyer with a verifiable identity and a clean history. The trust band is the credit score for that market.

What sites are asking for, measured this morning

How much of this has reached robots.txt files in practice? On 30 October we fetched robots.txt from the top 2,000 domains of the Tranco list with a self-identified research user agent, one request per host. 1,130 served a readable file. The rest were API and CDN hosts with nothing to say, 434 names that did not answer at all, and 145 that answered 403 to a crawler that had introduced itself, which is its own small finding.

Two details matter more than the headline. First, purpose is already being split by hand: of the 208 sites that block GPTBot entirely, 21 name OAI-SearchBot and let it through, and 18 do the same for ChatGPT-User. Operators are saying "index me, cite me, do not train on me" with the only vocabulary they have, which is one user agent per purpose. Second, of the 71 sites with a Content-Signal line, 39 carry the identical search=yes, ai-train=no, use=reference, which is Cloudflare's managed default. The new vocabulary is arriving through the CDN, not through site owners' editors, and 35% of the files we fetched came back with a cf-ray header. That is the mechanism by which a scoreboard becomes the rules.

A fetch loop that passes

None of the four habits is difficult. They are difficult to keep when a crawl is late and a queue is full, which is why they belong in the fetcher rather than in a runbook. What follows is a 226-line reference loop in standard-library Python, quoted in the four parts that matter, run against a local origin that has opinions: a robots.txt that disallows /private/, an ETag on every page, and a 429 with Retry-After: 2 for any client above two requests per second.

Read the rules before the first request

RFC 9309 says a crawler that reads robots.txt must follow it, may cache it, and should not keep the cached copy for more than 24 hours. It also says what to do when the file cannot be read: a 4xx means there are no rules, a 5xx or a network failure means assume you are not allowed, and try again later. The cache below does exactly that, per host, and exposes Crawl-delay so the budget can use it.

def _load(self, scheme: str, host: str) -> urllib.robotparser.RobotFileParser | None:
    url = f"{scheme}://{host}/robots.txt"
    req = urllib.request.Request(url, headers={"User-Agent": self.user_agent})
    parser = urllib.robotparser.RobotFileParser(url)
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            parser.parse(resp.read().decode("utf-8", "replace").splitlines())
            return parser
    except urllib.error.HTTPError as e:
        if 400 <= e.code < 500:
            parser.parse([])
            return parser
        return None
    except Exception:
        return None

def allowed(self, url: str) -> bool:
    parts = urllib.parse.urlsplit(url)
    host = parts.netloc
    with self._lock:
        cached = self._cache.get(host)
        if cached is None or time.time() - cached[0] > ROBOTS_TTL:
            cached = (time.time(), self._load(parts.scheme, host))
            self._cache[host] = cached
    parser = cached[1]
    return parser is not None and parser.can_fetch(self.user_agent, url)

The first time we ran the demo, the test server had not finished starting. Every URL came back disallowed, including the ones the rules would have allowed, because the loop could not read the rules. That is the correct outcome. A crawler that cannot find out what a site wants should not guess in its own favour.

Treat a 429 as a contract

The per-host budget is a token bucket, which is unremarkable. What matters is what it does when a 429 arrives. It honours Retry-After if there is one, backs off with jitter if there is not, halves its rate, and, the part most implementations skip, lowers its ceiling to 80% of the rate that drew the 429. The site has just told you its limit. Probing back up to it every few minutes is the "repeated pattern" the trust band is built to catch. Five 429s inside a minute open a breaker and the host is left alone for five minutes, which is when a human should be reading the log.

def on_429(self, retry_after: float | None) -> float:
    """Honour Retry-After when given, else back off exponentially with jitter."""
    with self._lock:
        now = time.monotonic()
        self._recent_429 = [t for t in self._recent_429 if now - t < self.trip_window] + [now]
        # The rate that drew the 429 is the host's limit: cap recovery
        # below it so one 429 is a lesson, not a cycle.
        self._ceiling = max(self.rate * 0.8, 0.05)
        self.rate = max(self.rate / 2, 0.05)
        if len(self._recent_429) >= 5:
            self._open_until = now + self.cooldown
            return self.cooldown
        if retry_after is not None:
            pause = retry_after
        else:
            pause = min(60.0, 2 ** len(self._recent_429)) * random.uniform(0.5, 1.5)
        self._open_until = now + pause
        return pause

Ask whether the page changed before asking for the page

Half of AI crawl traffic re-fetching unchanged pages is a number Cloudflare has a commercial interest in, but it is not implausible, and the fix predates the problem by twenty-five years. Remember the validators a page sends, send them back, and accept a 304 as the answer. The store below is deliberately naive; in production it is a column in the URL table.

class ConditionalStore:
    """Remember validators per URL so a re-crawl asks 'has this changed?' first."""

    def __init__(self):
        self._validators: dict[str, dict[str, str]] = {}

    def headers_for(self, url: str) -> dict[str, str]:
        v = self._validators.get(url, {})
        out = {}
        if "etag" in v:
            out["If-None-Match"] = v["etag"]
        if "last-modified" in v:
            out["If-Modified-Since"] = v["last-modified"]
        return out

    def remember(self, url: str, headers) -> None:
        v = {}
        if headers.get("ETag"):
            v["etag"] = headers["ETag"]
        if headers.get("Last-Modified"):
            v["last-modified"] = headers["Last-Modified"]
        if v:
            self._validators[url] = v

Write down every decision

The fetch method ties the three together and records each outcome as a line of JSON: what was asked for, what the rules said, what the site answered, how long the loop waited and what rate it was running at. When Open3DLab published its logs, the most damaging line was that the crawler "did not reduce its download attempts when faced with our blocks". An audit log is how you make the opposite claim and be believed. It is also, not by coincidence, what the EDPB asked for in the web-scraping guidelines whose consultation closed today: reliable sources, a timestamp per record, and validation before use.

def fetch(self, url: str) -> tuple[str, int, bytes | None]:
    """Return (outcome, status, body). Outcomes: fetched, unchanged, disallowed, throttled, error."""
    if not self.robots.allowed(url):
        self._record(url=url, outcome="disallowed")
        return "disallowed", 0, None
    budget = self._budget(url)
    waited = budget.acquire()
    req = urllib.request.Request(url, headers={"User-Agent": self.user_agent, **self.store.headers_for(url)})
    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            body = resp.read()
            self.store.remember(url, resp.headers)
            budget.on_success()
            self._record(url=url, outcome="fetched", status=resp.status, bytes=len(body), waited=round(waited, 2))
            return "fetched", resp.status, body
    except urllib.error.HTTPError as e:
        if e.code == 304:
            budget.on_success()
            self._record(url=url, outcome="unchanged", status=304, waited=round(waited, 2))
            return "unchanged", 304, None
        if e.code == 429:
            pause = budget.on_429(parse_retry_after(e.headers.get("Retry-After")))
            self._record(url=url, outcome="throttled", status=429, retry_after=e.headers.get("Retry-After"), pause=round(pause, 2), rate=round(budget.rate, 3))
            return "throttled", 429, None
        self._record(url=url, outcome="error", status=e.code)
        return "error", e.code, None

What the trace looks like

Two runs against the local site. The polite one at one request per second fetches three product pages, is refused one path by the rules without a packet leaving the box, then re-fetches the three pages and gets three 304s. The impatient one starts at five requests per second against a limit of two.

polite pass 1: ['fetched', 'fetched', 'fetched', 'disallowed']
polite pass 2: ['unchanged', 'unchanged', 'unchanged'] (4.0s for 7 fetches)
impatient run: ..T..T..T...............................
               3 x 429 in 40 tries, 33.2s, 1.11 pages/s achieved
rate trajectory: [5.0, 2.76, 0.76, 0.92, 1.12, 1.22, 1.22, 1.22, 1.22, 1.22]

{"url": ".../private/report", "outcome": "disallowed"}
{"url": ".../products/1", "outcome": "unchanged", "status": 304, "waited": 1.0}
{"url": ".../catalog/2", "outcome": "throttled", "status": 429, "retry_after": "2", "pause": 2.0, "rate": 2.5}
{"url": ".../catalog/8", "outcome": "throttled", "status": 429, "retry_after": "2", "pause": 2.0, "rate": 0.76}
{"url": ".../catalog/39", "outcome": "fetched", "status": 200, "bytes": 67, "waited": 0.82}

naive run:     ..T..T..T..T..T..T..T..T..T..T..T..T..T.

Three 429s, each honoured for the two seconds the site asked, each halving the rate and lowering the ceiling. After the third, 31 consecutive fetches with no throttling, settled at 1.2 requests per second under a limit of 2. The common retry loop, which waits out Retry-After and then carries on at the same rate, drew 13 429s in the same 40 requests, one every third request, and still fetched fewer pages per second: 0.94 against 1.11. Over a day that is thousands of entries in somebody's trust history, every one of them the repeated pattern the band is built to catch.

What this changes if you crawl through Crawlbase

Crawlbase does not read robots.txt on your behalf and will not pretend to. The policy is yours, and so is the history. What the Crawling API does is make the site's side of the conversation visible so your loop can act on it. Every response carries the target's own status in original_status and Crawlbase's verdict in cb_status: a site that answers 403 comes back as 705, a site that answers 429 as 706, and neither is charged, so backing off costs nothing but time. With get_headers=true the site's response headers come back too, which is where Retry-After, ETag and Last-Modified live. The budget and the validator store above work unchanged with those fields in place of the raw response.

On the Enterprise Crawler, the concurrency limit (20 by default) and the push rate (30 URLs per second per token) are the per-host budget's upper bounds, not a substitute for it. Run the loop above as the thing that decides what goes into the queue, and let the crawler be the thing that fetches it.

The practical position for the next year

There is a counter-argument, and it comes with data. On 20 October a developer who sells scrapers on Apify published the telemetry of 169,941 runs of his scrapers and found that calls arriving through MCP, that is from AI agents, succeeded 86.9% of the time against 47.9% for humans in the console, and that 0.6% of them tripped his abuse gate. "I had the story completely backwards," he wrote. Agents that call an API are the best-behaved clients on the web, because the API tells them the rules in a form they can read. The Roblox crawler misbehaved on sites that had no such interface, only a text file it chose not to parse.

That is the whole argument in one contrast. The web is not closing to crawlers. It is replacing a door with a ledger, and a ledger rewards exactly one thing: being legible. Sign your requests if you can, read the preferences whether or not you agree with them, let a 429 set your rate, send validators, and keep the log. Do that, and the trust band is a free asset that compounds. Skip it, and every crawl you run is evidence for the other side, timestamped by someone else.

Conclusion

October 2026 was the month crawler conduct acquired a record. Cloudflare began scoring Verified bots on robots.txt consistency and 429 response, Google started signing an agent that skips robots.txt, and a crawler that did neither ended up on the front page with its logs attached. The scoring is early, the vocabulary is still a draft, and only a third of the biggest sites have written any AI directive at all. But the direction is set, and the engineering it asks for is twenty-year-old HTTP done carefully: read the rules first, spend from a budget, obey the 429, validate before you fetch, and write everything down. If you run a crawler, run the loop above through Crawlbase and read its log before anyone else has to.

Frequently Asked Questions (FAQs)

Does the trust band apply to a crawler that is not a Verified bot?

No. The scoring attaches to an identity, so unsigned and unverified traffic is not scored; it is handled by the site's bot rules as before. The practical effect is the same: a crawler with no identity has no history to earn, so it cannot move out of the default-deny class.

Is a 429 ever safe to retry immediately?

Only when the response says so, and it never does. A 429 with Retry-After gives you a number; honour it. One without tells you to back off exponentially and lower your steady rate. Retrying the same request at the same concurrency is what the score is designed to catch.

Do I have to obey Content Signals and the aipref vocabulary?

Both are preferences, not access controls, and neither is a published standard yet. Cloudflare's crawl endpoint already enforces the use level, and the trust band reads consistency with a site's "published preferences", which will grow to include them. Reading them now costs nothing and starts the history early.

How is this different from what Google and Bing have always done?

Googlebot has read robots.txt and backed off on errors for two decades, and that conduct was never measured by anyone but Google. The change is that a third party now observes behaviour across millions of sites, publishes the result, and lets site owners act on it in their edge rules.

What does Crawlbase charge when a site rate-limits me?

Nothing. A target that answers 429 comes back as cb_status 706, a 403 as 705, and neither counts as a successful request, so backing off and retrying later costs only time.

VP

Viktor Petrov

Data Science Architect · Crawlbase

Data Science Architect at Crawlbase. Works on the models that keep large-scale crawling reliable against modern bot defenses.