For twenty years, a crawler's identity has been a claim. You write User-Agent: Googlebot in a header and the server decides whether to believe you. Because anyone can write anything there, operators learned to check the claim against circumstantial evidence: does a reverse DNS lookup on the connecting IP land in the right domain, does that domain forward-resolve back to the same address, is the IP inside a published range.
That check has held up the entire economy of who gets to crawl. It is also the reason bot detection turned into fingerprinting, behavioural scoring, and challenge pages, because circumstantial evidence is exactly as good as an attacker's willingness to rent the right address.
In 2026 that is being replaced. Not by better fingerprinting, but by asymmetric cryptography: a crawler signs its requests with a private key, publishes the public half at a well-known URL, and the server verifies a signature instead of weighing a hint. The claim becomes a proof.
The interesting part is not the cryptography, which is ordinary. It is what happened on the day it shipped.
- Web Bot Auth signs requests with Ed25519 and publishes verification keys at a well-known JWKS directory, so identity is verified rather than inferred.
- Cloudflare shipped it into the Verified Bots Program on 1 July 2025, the same day it began blocking AI crawlers by default and opened a pay-per-crawl beta.
- Identity became provable and permission became sellable on the same day. That is not a coincidence, it is the point.
- As of 18 August 2026 the protocol is still an individual Internet-Draft, not adopted by a working group, while gatekeepers verify it in production.
- Access is splitting into a signed lane and an unsigned lane, and no amount of fingerprint work gets you a key.
What the mechanism actually is
Web Bot Auth is a thin layer on top of RFC 9421, the HTTP Message Signatures standard. The bot holds an Ed25519 key pair. Every request carries three headers: Signature-Input describing what was covered, Signature carrying the bytes, and Signature-Agent pointing at the HTTPS location where the public keys live.
The signature covers request components rather than the whole body. The draft requires the signer to include either @authority or @target-uri, so a captured signature cannot be replayed against a different host or path.
GET /article/12345 HTTP/1.1 Host: example.com Signature-Agent: "https://crawler.example/.well-known/http-message-signatures-directory" Signature-Input: sig=("@authority" "signature-agent"); created=1740000000; expires=1740086400; keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"; tag="web-bot-auth" Signature: sig=:TUAfxGCoruBcOaEBRTdd6DcH0GJPd1v/1Vg1nCLmYFI=:
Four details in that block carry the design.
The keyid is not a name, it is a fingerprint. It is the base64url-encoded SHA-256 thumbprint of the JSON Web Key. You cannot claim someone else's identifier, because the identifier is derived from the key you must prove you hold.
The tag is web-bot-auth. Signatures are scoped to a purpose, so a signature minted for one protocol cannot be repurposed to satisfy another verifier.
Expiry is short. The draft recommends no more than 24 hours, which bounds how long a captured signature is worth anything.
The directory is discoverable. Keys are published as a JWKS at /.well-known/http-message-signatures-directory, so a verifier fetches and caches them without any prior relationship with the crawler.
The day identity and permission shipped together
On 1 July 2025 Cloudflare added message signatures to its Verified Bots Program, publishing Rust and TypeScript libraries and telling operators that applications with well-formed signatures would be approved faster.
On the same day, it began blocking AI crawlers by default for new domains and opened a private beta of pay-per-crawl, where a publisher sets a price and a crawler either pays it or does not get the page. Cloudflare sits in front of roughly a fifth of the web, so a default is close to a rule.
Read those two announcements as one and the shape is obvious. Charging for access requires knowing who is asking, and knowing who is asking requires an identity that cannot be forged. Pay-per-crawl is not possible on top of User-Agent. It becomes possible the moment identity is a signature.
Shipped before it was standardised
Here is the part worth sitting with. The protocol draft, draft-meunier-webbotauth-httpsig-protocol-02, is dated 18 August 2026 and is still an individual Internet-Draft. It is not working-group adopted, and it carries the standard notice that a draft is a work in progress.
Cloudflare's own documentation is candid about this: it says the implementation relies on IETF drafts and names the specific draft versions it tracks. Meanwhile the verification is live, and the same mechanism is being checked by other large operators in production.
That ordering is not unusual for the web, and it is not a scandal. It does have consequences you should plan around. A specification that is being implemented while it moves means the details move too: header shapes, directory paths, and tag values have all shifted between draft revisions. If you build against it, you are tracking a moving target maintained largely by the parties who benefit most from the outcome.
What this changes if you crawl
The arms race gets a second track, not a replacement
Nothing about signed agents removes fingerprinting. The machinery described in our anti-bot breakdown does not go anywhere: TLS fingerprints, header ordering, and behavioural scoring all still run, and they still decide what happens to unsigned traffic. What changes is that a signed, registered agent can skip most of it, because the expensive question of "is this who it says it is" has already been answered cheaply.
So the population splits. Signed agents get a fast path. Everyone else meets the same escalating wall as before, now with less of the operator's attention spent on distinguishing among them.
Identity is not authorisation
A verified signature proves who is asking. It says nothing about whether they may have the page. Those are separate decisions, and conflating them is the most common misreading of this technology. A publisher can verify your identity perfectly and still refuse you, charge you, or serve you a reduced version. Web Bot Auth makes the refusal precise rather than probabilistic.
A key is an allowlist entry, and allowlists have owners
In practice, being verified means being recognised by a gatekeeper, and the operators running those programmes are the same ones already deciding what reaches an origin. The registration path runs through the operator's programme, and approval is theirs to grant. That is a meaningful change in who decides which automated traffic is legitimate, and it moves that decision further from the publisher whose content is at stake and closer to the infrastructure in front of them.
It does not prove intent, respect for a licence, or good behaviour. It does not make a crawler polite, and it does not establish a right of access. It proves that the holder of one specific private key made one specific request before a deadline. Every other question the industry cares about is still argued in policy, contracts, and courts, and a signature only makes clear who the argument is with.
The practical position for the next few years
If you operate a large, named crawler with a public identity to defend, getting signed is straightforward and worth doing: generate a key, publish a directory, register with the programmes that matter for your targets.
If you are collecting public data at ordinary scale and you are not a household name, the honest reading is that the signed lane is not open to you in any practical sense, and the unsigned lane is getting narrower. Between those two facts, the cost of getting a page reliably keeps drifting away from clever client-side work and toward infrastructure that already solves access.
Access as a solved problem rather than a research project: rotating residential IPs, real browser rendering, and challenge handling inside the fetch, returning one clean response. Failed requests are not billed, so a hardening target costs you latency instead of invoice. Start free with 1,000 requests, no card.
Conclusion
The web is growing an identity layer for automated traffic, and it arrived in the ordinary way: a draft, a vendor implementation, and a business model that needed it. The cryptography is unremarkable. The consequence is not.
For two decades, the question at the edge was "does this look like a bot?" It is becoming "is this bot the one it says it is, and do we have an arrangement?" That is a better question, honestly asked. It is also a question that only has a good answer if you are already somebody, and that is the part the industry has not finished arguing about.
Watch the draft rather than the coverage. When it is adopted by a working group, when a second independent verifier ships, and when the first publisher sets a price that a crawler actually pays, the shape of the next few years will be visible in those three events.
Frequently asked questions
Is Web Bot Auth an approved internet standard?
Not yet. As of 18 August 2026 the protocol document is draft-meunier-webbotauth-httpsig-protocol-02, an individual Internet-Draft rather than a working-group adopted one, and Internet-Drafts are explicitly works in progress. The underlying signature format it builds on, RFC 9421, is a published standard. The layer on top that says what bots should sign and how verifiers should discover keys is still moving.
How is this different from checking reverse DNS?
Reverse DNS verification asks whether the connecting address belongs to the organisation the crawler claims. It is inference from network ownership, so it breaks when addresses are rented, delegated, or reassigned, and it cannot travel across networks. A signature is evidence about the request itself: only the holder of the private key could have produced it, from any address.
Does a signature let a crawler bypass anti-bot systems?
Only in the sense that it answers a question those systems exist to guess at. A verified agent still faces the operator's policy, which may be to allow, throttle, charge, or refuse. Signing removes ambiguity about identity, not the operator's right to say no.
Can a signature be replayed?
Not usefully, if it is built correctly. The signer must cover @authority or @target-uri, so a captured signature does not transfer to another host, and the draft recommends an expiry of no more than 24 hours, which bounds the window in which a replay is worth attempting at all.
What happens to crawlers that are not signed?
The same thing that happens today, with less patience. Unsigned traffic keeps meeting fingerprinting, behavioural scoring, and challenges, and as more legitimate automated traffic moves into the signed lane, the unsigned population looks more uniformly suspicious to the systems judging it.
Should I implement this now?
If you run a named crawler whose identity is part of your product, yes: the mechanics are a key pair and a JSON document, and being early is cheap. If you are consuming public data at ordinary scale, implementing the client side gains you nothing on its own, because the value is in being recognised by a programme rather than in producing a valid signature. Track the draft and watch which verifiers ship.
Crawl any site at scale, without fighting infrastructure.
Crawlbase handles proxies, fingerprints, and CAPTCHAs so your team ships data pipelines instead of maintaining crawl plumbing. Up to 5,000 requests free, no card required.