A scraper that works once has proved exactly one thing: that one request, from one exit, at one moment, came back with content. It has not told you how often that target answers, how long it takes, what a page costs, whether a browser is doing any work, or which of the site's URL shapes the failures cluster around. Those are the numbers that decide whether a pipeline is worth building, and they are usually discovered after the pipeline exists.
The Crawlbase Web Scraping Cookbook publishes them up front. It is one page per site, and every page is measured on real Crawlbase traffic before anything is written: success rate, median answer time, credit price, how much of the successful traffic used a browser, which country route the winning calls set, which URL patterns account for the successes, and what the failures actually were. It currently covers 17,739 sites measured and 356 recipes, on August 2026 numbers, refreshed every month.
- A success rate on its own is not a profile. Two sites at 98% can differ by 20x in latency and 20x in price.
- Browser rendering is a measurement, not a default. On four of the six sites below, the plain token wins and a browser buys nothing.
- Country routing is the same kind of question. It moves Trustpilot by four points and does nothing at all for the others.
- Failure classes are instructions. A 429 means pace the calls, a 403 means fix the exit, a 5xx means wait.
- The numbers describe a measured window, not a guarantee, which is why they are restated every month.
What a recipe measures
Each page is built from observed traffic to that site, not from a test run performed for the page. The measurements are the ones a pipeline actually has to plan around.
| Measurement | What it answers | What it changes |
|---|---|---|
| Success rate | How often calls to this site come back with content | Retry budget and whether the target is worth a pipeline |
| Median answer | How long a typical call takes | Concurrency, scheduling, client timeouts |
| Credit price | What one page costs, by domain complexity, doubled on a JavaScript fetch | Unit economics before scale, not after |
| Browser share | How much of the winning traffic used the JavaScript token | Whether rendering is required or just expensive |
| Country | Which exit the successful calls used | Whether country=XX earns its place |
| URL patterns | Which page shapes the successes are concentrated in | What to implement and validate first |
| Failure classes | What the site answered when it refused | Retry logic that matches the refusal |
One detail in the success counting is worth stating plainly, because it changes what the rate means. A site that answers 200 with a block page or a consent wall has not returned content, so the Cookbook counts it as a failure rather than a success. On Google, that class is the single largest failure bucket at 33.8%.
Six sites, six different shapes
The fastest way to see why a single number is not enough is to put six real recipes next to each other. All figures below are the August 2026 measurements published on each page, with patterns and failures read from the request log since 14 August 2026.
Google: fast, cheap enough, no browser
The google.com recipe measures 97.0% success, a 1.9-second median answer, and 2 credits per page. Plain-token calls succeed at 96.9% and carry 99.8% of the traffic, so a browser is pure cost here. Search pages dominate: /search is 98.3% of successful requests, and 6.3% of calls ask for the google-serp scraper to get structured JSON instead of HTML. The failures split almost evenly between block or consent pages answered as 200 (33.8%) and requests that never answered inside the timeout (33.5%), with rate limits at 23.5%.
LinkedIn: reliable, slow, and the expensive one
The linkedin.com recipe measures 98.0% success, a 14.7-second median answer, and 20 credits per page, the Extreme II price tier. Plain-token calls succeed at 99.9% and make up 96.0% of traffic. The successes concentrate in job pages (41.5%), company pages (27.3%) and profiles (18.5%). Failures are dominated by the site's own refusal status, 999, at 58.5%, with 591 responses at 27.1%.
LinkedIn is the clearest argument for reading the whole profile. Judged on success rate it is the second most reliable site in this group. Judged on cost per page it is twenty times Target, and judged on latency it is nine times Google.
Allegro: near-perfect, and in no hurry
The allegro.pl recipe measures 99.7% success, a 16.5-second median answer, and 2 credits per page. All of the successful traffic uses the plain token. Product pages under /produkt/{slug} account for 77.3% of successes. Failures are led by timeouts (44.3%), then 403 refusals (36.9%) and 400 responses (12.8%).
Target: the cheap, fast one that gets rate limited
The target.com recipe measures 96.4% success, a 1.6-second median answer, and 1 credit per page. Plain-token calls succeed at 97.7% and carry 97.2% of traffic, and two product URL shapes cover 99.3% of successes between them. The failure profile is the interesting part: 48.1% are 429 rate limits, followed by 403 at 32.4% and timeouts at 13.3%. Cheap and fast, but the site pushes back on pace, which is a scheduling problem rather than a configuration one.
Trustpilot: the one where rendering and country both matter
The trustpilot.com recipe measures 90.9% success, a 32.0-second median answer, and 1 credit per page, 2 credits on the JavaScript token this recipe needs. This is the site in the group where configuration changes both the outcome and the bill. JavaScript-token calls succeed at 94.0% against 29.5% for the plain token, and 94.0% of calls use it. Country routing moves the result too: 46.2% of successful calls set the US exit and succeed at 92.4%, against 88.2% with no country set. Reviews under /review/{slug} are 97.5% of successes. Failures are led by 403 at 45.9%.
That is the call the recipe recommends, and it is the one worth copying, because it carries both decisions at once:
curl "https://api.crawlbase.com/?token=YOUR_JS_TOKEN&country=US&url=https%3A%2F%2Fwww.trustpilot.com%2Freview%2Fsd.se"
QQ.com: almost perfect, and it needs the browser anyway
The qq.com recipe measures 99.9% success, a 6.1-second median answer, and 1 credit per page, 2 credits on the JavaScript token this recipe needs. It is the most reliable site in this group and it still needs rendering: JavaScript-token calls succeed at 100% against 86.8% for the plain token, and 98.9% of calls use it. Tag pages are 99.8% of successes. Most failures are the site's own 5xx responses, 72.4% together.
QQ.com and Trustpilot make the same point from opposite ends. A headline success rate does not tell you whether a browser is required, because the rate already reflects the fact that callers are using one.
The six side by side
| Site | Success | Median answer | Price | Browser | Country | The shape |
|---|---|---|---|---|---|---|
| google.com | 97.0% | 1.9s | 2 credits | No | None | Fast and simple |
| linkedin.com | 98.0% | 14.7s | 20 credits | No | None | Reliable, slow, expensive |
| allegro.pl | 99.7% | 16.5s | 2 credits | No | None | Reliable but slow |
| target.com | 96.4% | 1.6s | 1 credit | No | None | Cheap, fast, rate limited |
| trustpilot.com | 90.9% | 32.0s | 2 credits on JS | Yes | US helps | Configuration decides it |
| qq.com | 99.9% | 6.1s | 2 credits on JS | Yes | None | Reliable, rendering-dependent |
Reading a recipe before you write the scraper
The practical use of the profile is that it removes the guessing round at the start of an integration, where most of the wasted effort is.
Start on the configuration that already wins
Every recipe names the token and country that the successful traffic used, so the first call is an informed one rather than a probe. For a plain-token site like Google, that is the whole configuration:
from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) r = api.get('https://www.google.com/search') html = r['body']
Add a browser only where the measurement asks for it
Rendering is the most common reflex and the most common waste. Google, LinkedIn, Allegro and Target are all handled on the plain token in the observed traffic, so a browser there adds cost and latency to a call that already worked. Trustpilot and QQ.com are the opposite case, where rendering is the difference between 94.0% and 29.5%, or between 100% and 86.8%. On the Crawling API you can ask for it two ways: call with the JavaScript token, or call with the Normal token and add javascript=true, which switches to your JavaScript key before the crawl so one key covers everything. With Smart AI Proxy the switch is javascript=true in the CrawlbaseAPI-Parameters header. Either route is billed as a JavaScript request, which is where the doubled credit price on those two sites comes from.
Let the failure class choose the retry
The failure breakdown is the part that pays off after launch, because each class calls for a different response. Retrying a 429 immediately reproduces it. Retrying a 403 without changing the exit reproduces that too.
- 429 is the site pacing you. Space the calls, or hand them to the Crawler, which paces them for you. Target's failures are half this.
- 403 is the exit being refused. Set the country the successful calls use and retry once; a second refusal usually means the URL itself is protected.
- 5xx is the site failing rather than the fetch. Wait and retry later. On QQ.com this is almost three quarters of all failures.
- Timeouts mean the page did not finish inside the limit. Keep your client timeout above the site's median answer, which is exactly why the recipe publishes it.
- A 200 carrying a block or consent page is a failure wearing a success status. It is not billed, and it is the reason to check content rather than status.
The recipes are measured on it: one endpoint, browser rendering and anti-bot handling inside the fetch, and cb_status telling you whether the page came back. Failed requests are not billed. Start free with up to 5,000 requests, no card.
From recipe to pipeline
The profile fits the early part of a build, where the unknowns are cheapest to remove.
For a team choosing infrastructure rather than writing the fetch, the same numbers answer a different question. Reliability, latency and price are per-target properties, not platform properties, so "can you scrape this site" is less useful than "what does this site cost, how slow is it, and what does it do when it refuses". The six sites above range from 1 to 20 credits a page and from 1.6 to 32.0 seconds, on one platform. Cost modelling starts from the target's price:
100,000 pages x 1 credit = 100,000 credits (target.com) 100,000 pages x 20 credits = 2,000,000 credits (linkedin.com)
The pricing estimator turns those credits into a monthly figure. The point is that the multiplier is knowable before the pipeline exists rather than after the first invoice.
What the numbers do not claim
A recipe describes measured traffic for a stated window. It is not a promise about tomorrow, and it is not a claim that every page on the site behaves like the ones in the sample. Sites change protection, rotate infrastructure and re-tune rate limits, which is why every page carries the month it was measured and a last-tested date, and why the whole set is refreshed monthly rather than left to age. When a site stops answering, its recipe is removed instead of being kept as documentation of something that no longer works.
The patterns and failure classes come from the request log since 14 August 2026, so they describe the page shapes that Crawlbase accounts actually fetch. A URL shape nobody requests will not appear, even if the site serves it.
Conclusion
Most scraping decisions get made twice: once from assumption, and again after the traffic shows what the site really does. The Cookbook moves the second version earlier. Before writing an extractor you can see how often the target answers, how long it takes, what a page costs, whether the browser is doing any work, which exit the winning calls use, which URL shapes carry the successes, and what the refusals were.
Find your target in the Cookbook, start from the call the measurements support, and create a free account to run it against your own workload.
Frequently Asked Questions (FAQs)
Where do the Cookbook numbers come from?
From Crawlbase traffic to that site: every request from every account over the stated month for the success rate, and the request log since 14 August 2026 for patterns, countries and failure classes. They are measurements of observed traffic for that window, not guarantees of future performance, and no customer or account is identified anywhere in the data.
Does "browser needed" mean I have to run a browser myself?
No. It means the target needs JavaScript rendering, and Crawlbase does the rendering. With the Crawling API and the Crawler you ask for it either with the JavaScript token or with your Normal token plus javascript=true, which switches keys before the crawl; with Smart AI Proxy you pass javascript=true in the CrawlbaseAPI-Parameters header. Both count as a JavaScript request for billing.
Can I estimate what a target will cost before I build?
Yes, and that is most of the point. The recipe publishes the credit price for a page on that site, set by its domain complexity, so pages times price gives you the credit figure and the estimator converts it. Mind the browser column while you do it: a site that needs rendering bills the doubled JavaScript figure, which is why Trustpilot and QQ.com cost 2 credits a page rather than 1. Actual consumption still depends on your configuration and which product you send the traffic through.
Why is a 200 response sometimes counted as a failure?
Because a block page or a consent wall served with a 200 is not the content you asked for. Counting it as success would inflate every rate on the site and hide the most common way modern targets refuse. Those responses are not billed either.
How often is a recipe refreshed?
Monthly. Each page states the month behind its measurements and the date it was last tested, so a stale profile is visible rather than silent. Recipes for sites that stop answering are removed rather than left in place.
Crawl any site at scale, without fighting infrastructure.
Crawlbase handles proxies, fingerprints, and CAPTCHAs so your team ships data pipelines instead of maintaining crawl plumbing. Up to 5,000 requests free, no card required.