Most developers can get an AI agent to scrape a single page with a well-phrased prompt. Ask Claude to grab the price from this URL, paste a link, and you usually get something usable in one turn.
That workflow breaks quickly once you scale past a prototype, because the moment you move from one page to ten thousand, you stop fighting prompts and start fighting infrastructure. The same failures show up again and again in large-scale retrieval testing:
- Agents consuming CAPTCHA pages as if they were real content.
- 200 KB HTML payloads exploding token usage.
- Retries multiplying inference costs.
- Frontend redesigns silently breaking extraction logic.
- The same URLs getting re-crawled over and over.
Most AI agent failures are infrastructure failures wearing a prompting costume. This guide is not about better prompts. It is about building the layer between the live web and your model: normalized Markdown retrieval through the Crawlbase Web MCP Server, circuit breakers that reject poisoned responses before inference, and Cloud Storage as durable agent memory. Every example below has a runnable counterpart in the companion repository.
- Most AI scraping demos fall apart once they scale beyond a few pages.
- Raw HTML acts as context poison for LLMs and inflates token costs.
- The Web MCP Server returns clean Markdown instead of bloated frontend markup.
- Circuit breakers stop poisoned retrieval before it reaches the model.
- Persistent Cloud Storage turns agents into long-running monitoring systems instead of stateless crawlers.
Why AI agents break at scale
AI agents fail at scale because the public web is noisy, unstable, and expensive for a language model to process directly. A modern webpage is not just content. It is JavaScript bundles, hydration payloads, analytics scripts, duplicated DOM trees, cookie banners, and framework state. A pricing page that looks simple can return well over 200 KB of raw HTML before any normalization, and most of that has nothing to do with what the agent actually needs.
Plenty of agent pipelines still dump that entire response straight into the context window, which creates two distinct infrastructure problems.
The token tax is the cost of carrying irrelevant frontend markup through inference. Every unnecessary retrieval byte eventually converts into token cost, latency, retry overhead, and memory pressure. Context collapse is the reasoning failure that follows: the model loses semantic focus because it is busy processing markup instead of meaning.
In practice this produces failures that look almost comical in isolation and expensive in aggregate. Agents extract prices from recommendation carousels, summarize cookie banners as page content, mistake hydration JSON for product data, or treat a CAPTCHA interstitial as a valid response.
The most dangerous production pattern is that many of these failures still return HTTP 200. A soft block, a challenge page, or an empty hydration shell arrives with the same status code as a perfect crawl. If your pipeline treats 200 as "good data," it has no failure signal at all.
Why Markdown beats raw HTML
Markdown reduces token usage and improves extraction reliability because it removes frontend noise while preserving semantic structure. Raw HTML carries JavaScript payloads, CSS classes, navigation markup, tracking scripts, and framework artifacts that the model does not need. Markdown keeps what reasoning actually depends on: headings, paragraphs, lists, tables, and links.
| Input format | Approx payload | Estimated tokens | Extraction reliability |
|---|---|---|---|
| Raw HTML | 200 KB | 50K+ | Unstable |
| Markdown | 10–30 KB | 2K–7K | Significantly cleaner |
The exact reduction varies by site, but the direction is consistent: an order of magnitude less payload for the same information. That shows up as lower token costs, faster inference, cleaner extraction, and more stable reasoning, all from improving the signal-to-noise ratio inside the context window.
This is why the Web MCP Server exposes normalized retrieval through crawl_markdown, the same mechanism behind format=md in the Crawling API. With normalization handled upstream, the prompt gets to be short and specific:
Use crawl_markdown on https://example.com/pricing and return only: plan names, monthly prices, and currency. Do not paste raw HTML. If the crawl fails, report the tool error.
The principle underneath: a model should never consume arbitrary internet responses directly. It should consume normalized semantic representations.
What the agent data plane is
The agent data plane is the infrastructure layer between the internet and the LLM. Most teams spend their attention on prompts, models, and agent frameworks, but in production the bigger bottleneck is almost always retrieval reliability. The data plane owns retrieval, validation, normalization, persistence, and context governance, and it does all of that before a single byte reaches the reasoning layer.
Without that layer, agents operate directly against unstable live webpages. It demos beautifully and degrades fast.
| Prototype agent | Production agent |
|---|---|
| Reads raw HTML directly | Reads normalized Markdown |
| Re-crawls on every request | Queries storage first |
| Blind retries | Circuit breaker policies |
| Stateless | Persistent memory |
| Large prompt payloads | Context-governed retrieval |
As AI systems scale, the choke point moves away from prompting and toward retrieval reliability, memory management, context efficiency, and deterministic state. That is exactly why Markdown normalization, retrieval validation, and persistent storage stop being optimizations and start being architecture.
Circuit breakers: validate before inference
A circuit breaker keeps poisoned retrieval out of the context window, and in production scraping that matters more than it sounds, because bad retrieval is usually worse than missing retrieval. Missing data announces itself. Bad data gets summarized, stored, and acted on.
The companion circuit_breaker.py module implements fail-closed validation ahead of the reasoning layer:
def evaluate( result: CrawlResult, *, max_body_chars: int = DEFAULT_MAX_BODY_CHARS, expect_markdown: bool = True, ) -> CircuitDecision: if result.http_status != 200: return CircuitDecision(False, f"HTTP status {result.http_status}") if result.cb_status and result.cb_status != "200": return CircuitDecision( False, f"Crawlbase cb_status={result.cb_status}" ) if not result.body.strip(): return CircuitDecision(False, "empty body") if expect_markdown and not result.content_type.startswith("text/markdown"): return CircuitDecision( False, f"expected markdown, got {result.content_type}" ) if len(result.body) > max_body_chars: return CircuitDecision( False, f"body exceeds {max_body_chars} chars" ) return CircuitDecision(True, "ok")
That function is a firewall between the public web and the context window. Rather than trusting every response, it checks transport status, Crawlbase cb_status, empty bodies, content type, and payload size before anything reaches the model. Note that cb_status is the current name for the Crawlbase status header; older code and tutorials may still call it pc_status.
This matters because anti-bot systems rarely fail loudly. A request can succeed at the transport layer while serving a CAPTCHA page, an access-denied document, a challenge interstitial, or an empty hydration shell. Models will reason confidently over all of those unless the retrieval layer rejects them first.
Without a circuit breaker, the production failure chain is predictable:
- The target site returns a CAPTCHA page.
- The agent treats that HTML as real content.
- Extraction pipelines store poisoned data.
- Monitoring systems report changes that never happened.
- Retry loops multiply both token and crawl costs.
One bad retrieval can poison an entire multi-agent workflow, which is why validation belongs before inference and not after it. The runnable version lives in the circuit breaker implementation.
Why storage beats re-crawling
Production agents should query memory before they query the internet. Most agent tutorials treat the web as stateless; real systems cannot afford to. Re-crawling the same URLs repeatedly increases token spend, destabilizes retrieval, produces inconsistent outputs, and adds crawl overhead that buys nothing.
This is where Crawlbase Cloud Storage earns its place in the architecture. The ingestion pipeline stores normalized Markdown snapshots with store=true instead of reprocessing live pages on every run:
params = { "token": token, "url": url, "format": "md", "md_readability": "true", "store": "true", }
The result is an rid-based retrieval workflow where agents exchange references instead of full payloads. Instead of pushing large documents through the model on every hop, the system persists normalized Markdown, retrieval metadata, rid references, and token estimates, then fetches by reference through storage_get when a step actually needs the content.
There is a second benefit that is easy to miss: determinism. The live internet is non-deterministic thanks to frontend redesigns, A/B tests, personalization, and ordinary DOM churn. Storage-backed snapshots give you a fixed input to reason over, which makes the difference between a pipeline you can debug and one you can only observe.
The full workflow lives in the companion storage-first ingestion script.
Change detection without reprocessing everything
Persistent storage makes change detection dramatically cheaper, because the system compares normalized snapshots instead of pushing whole webpages back through the model. The companion change_detection.py script compares stored Markdown against a fresh normalized crawl, and the comparison is intentionally boring:
prev_hash = hashlib.sha256(previous.encode("utf-8")).hexdigest() curr_hash = hashlib.sha256(current.encode("utf-8")).hexdigest()
If the two hashes match, the script reports UNCHANGED and stops. No summarization pass runs, no downstream inference happens, and no tokens are spent. That is the whole point of the pattern: the cheapest reasoning step is the one you never execute. The runnable version is the change detection implementation.
The companion data plane repository
The companion repository shows these patterns working together in one retrieval pipeline. Instead of sending raw webpages into the model, the workflow validates responses before inference, converts pages into normalized Markdown, stores snapshots for later retrieval, and compares stored against live content deterministically. Three modules carry it:
-
circuit_breaker.pyfor fail-closed retrieval validation. -
ingest.pyfor storage-first Markdown ingestion. -
change_detection.pyfor snapshot comparison and monitoring workflows.
The important part is not the scripts. It is the shape: a URL batch enters, the circuit breaker rejects what should never reach a model, survivors are normalized to Markdown and stored, and the agent later reads small slices by reference rather than re-crawling. Agents should reason over validated, normalized, persistent retrieval, not over arbitrary internet responses.
Connect the Web MCP Server to Claude
The Web MCP Server plugs directly into Claude Desktop or Claude Code. The AI and MCP documentation and the Claude integration guide cover setup, Cloud Storage workflows, and the retrieval tools in more detail. The Claude Desktop configuration is short:
{ "mcpServers": { "crawlbase": { "type": "stdio", "command": "npx", "args": ["@crawlbase/mcp@latest"], "env": { "CRAWLBASE_TOKEN": "YOUR_TOKEN", "CRAWLBASE_JS_TOKEN": "YOUR_JS_TOKEN" } } } }
After restarting Claude, the retrieval tools become available inside the model environment: crawl_markdown for normalized fetches, plus storage_get, storage_list, and storage_bulk_get for reading what you already crawled. The same infrastructure patterns then work interactively inside Claude or operationally through Python pipelines, which is the point of putting them in the data plane rather than in a prompt.
Production patterns that scale
A handful of rules separate retrieval systems that survive production from the ones that only demo well.
Default to Markdown
Treat raw HTML as the fallback, not the default reasoning format. Most pages carry far more frontend noise than content, and that noise adds token cost without improving extraction quality. Markdown keeps the semantics and drops the clutter, which makes reasoning faster, cheaper, and more stable.
Fail closed
Bad retrieval is worse than missing retrieval. A CAPTCHA page, a soft block, or a broken render can return HTTP 200 while feeding poison into the model. Reject suspicious retrieval at the boundary instead of hoping the model notices.
Query memory first
Check storage before re-crawling. Repeatedly fetching the same pages creates token spend, unstable outputs, and duplicated crawl overhead. Persistent snapshots let agents reason over a fixed input instead of a moving one.
Enforce context budgets
Context windows are an infrastructure resource with a price tag. Large payloads increase cost, latency, and reasoning instability together, so cap payload size before content enters the window rather than after the bill arrives.
Separate retrieval from reasoning
Validation and normalization belong upstream of the model. Splitting the pipeline into retrieval, validation, normalization, persistence, and reasoning gives you clean failure isolation, more stable outputs, and a system you can actually debug at scale.
What actually scales
Vibe coding is enough to get an agent working on one page. The hard part starts when the same workflow has to run across thousands of pages, continuously, without supervision. At that point the question stops being how to prompt the model and becomes how to stop feeding it noisy, unstable, or expensive retrieval.
The reliable systems tend to converge on the same set of choices: Markdown instead of raw HTML, storage instead of stateless crawling, circuit breakers instead of blind retries, deterministic snapshots instead of live browsing, and validation before inference. As agents scale, they behave less like chatbots and more like distributed systems, and the bottleneck shifts to retrieval reliability, memory architecture, normalization, persistence, and token governance.
If you are building in this direction, our walkthroughs on AI agent workflows with the Web MCP Server and building an AI research dataset show the same data plane applied to concrete jobs. The next generation of AI systems will not be defined only by better prompts. It will be defined by better retrieval infrastructure.
Give Claude and any other MCP client a production data plane in one tool call. Every crawl renders JavaScript behind a rotating residential IP and returns clean Markdown instead of 200 KB of frontend markup, with optional Cloud Storage so agents read by reference instead of re-crawling. Get your API tokens and build on the free tier.
Frequently asked questions
Why do AI agents fail when scraping large numbers of pages?
Small demos work because the model quietly compensates for noisy retrieval. At higher crawl volumes, agents start consuming oversized HTML payloads, CAPTCHA pages, duplicated frontend markup, and unstable live responses, which produces context collapse, inflated token costs, and inconsistent extraction. The failure is in the data plane, not the prompt.
Why is Markdown better than HTML for AI agents?
Markdown preserves semantic structure while removing most frontend noise, which reduces token usage, context pollution, inference latency, and extraction instability. The same page that costs 50K+ tokens as raw HTML typically fits in a few thousand tokens as Markdown, with better signal-to-noise inside the context window.
What is a circuit breaker in AI infrastructure?
It is a validation gate that runs before content reaches the LLM. It rejects HTTP errors, non-200 cb_status values, empty bodies, unexpected content types, and oversized payloads, so poisoned retrieval never becomes a reasoning input. The policy is fail closed: when a response looks wrong, drop it rather than summarize it.
Why should agents use storage instead of re-crawling pages?
Persistent storage gives you deterministic retrieval, historical comparison, replayable workflows, lower crawl cost, and lower token usage. Agents exchange rid references instead of full documents and fetch content only when a step needs it. Production agents should query memory before querying the internet.
What is the agent data plane?
The agent data plane is the infrastructure layer between the internet and the LLM. It handles retrieval, validation, normalization, persistence, and context governance, and it becomes more important the further an AI system scales past a prototype.
Crawl any site at scale, without fighting infrastructure.
Crawlbase handles proxies, fingerprints, and CAPTCHAs so your team ships data pipelines instead of maintaining crawl plumbing. Up to 20,000 requests free, no card required.