Cookbook·News and media·bbc.com
Scrape sport pages on bbc.com
The call that returns sport pages from bbc.com today, with the parameters, patterns and failure modes read off the request log since 14 Aug 2026.
97.9% success · Aug 2026Browser on 23% of callsStandard price · 1 credit
Measured on Crawlbase
Success rate over every request from every account in August 2026: 97.9%. Patterns, countries and failures come from the request log since 14 Aug 2026; 22.6% of successful calls used the JavaScript token and the median answer took 5.4s.
Among the 74 news and media sites in the Cookbook, bbc.com sits at 49 of 74 by success rate; the category median is 98.9%. Its median answer of 5.4s is slower than the category median of 2.3s. 58 of the 74 accept the plain token; this one does on some pages only.
The call
Article pages are the shape the accounts fetch most here, with the country and token that succeed most often. Replace the placeholder path with a real article URL from the site.
curl "https://api.crawlbase.com/?token=YOUR_TOKEN&url=https%3A%2F%2Fwww.bbc.com%2Fsport%2Fexample-page%2Fexample-page"from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) r = api.get('https://www.bbc.com/sport/example-page/example-page') html = r['body']
const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const r = await api.get('https://www.bbc.com/sport/example-page/example-page');
Parameters that matter
| Parameter | Set it to | Why |
|---|---|---|
country | leave unset | Most successful calls set no country (86.5%) and succeed at 99.9%. |
javascript | leave unset | Plain-token calls succeed at 100%, JavaScript-token calls at 98.3%. Both work; start without the browser and switch per URL pattern where the table below shows it. |
URL patterns accounts fetch
| Pattern | Share of successes | JavaScript token | Country | Query parameters |
|---|---|---|---|---|
/sport/{slug}/{slug} | 45.0% | 91.7% | none | page, xtor |
/news/{slug}/{slug} | 42.5% | 49.4% | none | none |
/ | 2.3% | 0.0% | none | none |
/news/{slug} | 2.0% | 75.0% | none | none |
/news | 2.0% | 0.0% | none | none |
Fields you get
An article page carries a headline, a date and an author, usually in an ld+json Article block, and the body in the HTML. The block often names the section too.
What breaks, and the fix
Of the failures since 14 Aug 2026, the site answered:
999: 66.7% of failures 999 is not a standard HTTP answer; the site sent it as a refusal. Treat it like a block: retry once with the JavaScript token and the country the successful calls use, then stop.
504: 16.7% of failures A 504 is a timeout: the page did not finish inside the limit. Retry once, and keep your own client timeout above the median answer time shown for this site.
591: 16.7% of failures A 5xx is the site failing, not the fetch. Retry after a minute.
Run it on a schedule
News moves by the hour. Poll the section or the sitemap for new links, then push the article URLs to the Crawler with a callback so the fetches are paced. The median answer on this site is 5.4s.
Questions
Do I need a browser to scrape bbc.com?
It depends on the page. Plain-token calls succeed at 100%, JavaScript-token calls at 98.3%; the URL pattern table shows which paths accounts fetch with a browser.
How much does a bbc.com page cost on Crawlbase?
1 credit without a browser. bbc.com is in the standard tier.
Which country should I set for bbc.com?
None is needed: most successful calls leave it unset.
Site names identify the pages a recipe fetches. Crawlbase is not affiliated with any site listed. Use recipes within the acceptable use policy.Measured numbers from August 2026, refreshed monthly