Scrape search results on google.com

About google.com

Its sitemaps list 3,142 URLs across 15 files.

Its robots.txt names 1 sitemap and disallows 12 paths, among them /search, /sdch, /groups.

Measured on Crawlbase

Success rate over every request from every account in August 2026: 97.0%. Patterns, countries and failures come from the request log since 14 Aug 2026; 0.2% of successful calls used the JavaScript token and the median answer took 1.9s.

Among the 11 tech and developer sites in the Cookbook, google.com ranks 7 of 11 by success rate; the category median is 97.8%. Its median answer of 1.9s sits at the category median of 1.9s. 8 of the 11 accept the plain token, this one among them.

The call

Search result pages are the shape the accounts fetch most here, with the country and token that succeed most often. Replace the placeholder query with your own.

curl "https://api.crawlbase.com/?token=YOUR_TOKEN&url=https%3A%2F%2Fwww.google.com%2Fsearch"

from crawlbase import CrawlingAPI
api = CrawlingAPI({'token': 'YOUR_TOKEN'})

r = api.get('https://www.google.com/search')
html = r['body']

const { CrawlingAPI } = require('crawlbase');
const api = new CrawlingAPI({ token: 'YOUR_TOKEN' });

const r = await api.get('https://www.google.com/search');

Parameters that matter

URL patterns accounts fetch

Fields you get

Structured data Crawlbase found on google.com, by page type. These are the schema.org types in the page, the fields your parser can read straight from the JSON-LD block:

• Other: FAQPage, VideoObject

What breaks, and the fix

Of the failures since 14 Aug 2026, the site answered:

Run it on a schedule

Search is the expensive side of a site. Keep the query list short, run it on a cadence through the Crawler with a callback, and fetch afterwards only the result pages you need. The median answer on this site is 1.9s.

Questions

Other pages on google.com

Pages google.com lists in its own sitemaps, fetched by Crawlbase on 6 Sep 2026 and verified in 2 consecutive runs. Each row is the cheapest call that returned the page with its content.