Product / Crawling APICrawling API.
Crawling API.
Any URL, fully rendered.
Send any URL and get the fully rendered HTML back, through 140M residential IPs with anti-bot handling built in.
Add a scraper or autoparse for structured JSON, or a screenshot.
Live crawl feed1.24M req/minStreaming
200target.com/p/-/A-79404211ES44ms
404indeed.com/jobs?q=developerFR81ms
200github.com/crawlbaseGB100ms
200ebay.com/itm/204512389011ES82ms
200google.com/search?q=web+scrapingDE210ms
200stackoverflow.com/questions/11227809GB136ms
200glassdoor.com/Reviews/index.htmDE161ms
200yelp.com/biz/blue-bottle-coffeeBR120ms
200tripadvisor.com/Restaurants-g60763US109ms
200zillow.com/homes/for_sale/SG45ms
301google.com/search?q=web+scrapingES41ms
200github.com/crawlbaseNL46ms
200google.com/search?q=web+scrapingNL47ms
200stackoverflow.com/questions/11227809GB139ms
200reddit.com/r/programmingNL113ms
200ebay.com/itm/204512389011NL197ms
200zillow.com/homes/for_sale/AU74ms
200glassdoor.com/Reviews/index.htmCA138ms
200amazon.com/dp/B08N5WRWNWBR124ms
200amazon.com/dp/B08N5WRWNWNL165ms
200zillow.com/homes/for_sale/IN171ms
200producthunt.com/posts/notionNL58ms
200zillow.com/homes/for_sale/IN91ms
301target.com/p/-/A-79404211JP61ms
200tripadvisor.com/Restaurants-g60763SG164ms
200tripadvisor.com/Restaurants-g60763CA202ms
200target.com/p/-/A-79404211ES44ms
404indeed.com/jobs?q=developerFR81ms
200github.com/crawlbaseGB100ms
200ebay.com/itm/204512389011ES82ms
200google.com/search?q=web+scrapingDE210ms
200stackoverflow.com/questions/11227809GB136ms
200glassdoor.com/Reviews/index.htmDE161ms
200yelp.com/biz/blue-bottle-coffeeBR120ms
200tripadvisor.com/Restaurants-g60763US109ms
200zillow.com/homes/for_sale/SG45ms
301google.com/search?q=web+scrapingES41ms
200github.com/crawlbaseNL46ms
200google.com/search?q=web+scrapingNL47ms
200stackoverflow.com/questions/11227809GB139ms
200reddit.com/r/programmingNL113ms
200ebay.com/itm/204512389011NL197ms
200zillow.com/homes/for_sale/AU74ms
200glassdoor.com/Reviews/index.htmCA138ms
200amazon.com/dp/B08N5WRWNWBR124ms
200amazon.com/dp/B08N5WRWNWNL165ms
200zillow.com/homes/for_sale/IN171ms
200producthunt.com/posts/notionNL58ms
200zillow.com/homes/for_sale/IN91ms
301target.com/p/-/A-79404211JP61ms
200tripadvisor.com/Restaurants-g60763SG164ms
200tripadvisor.com/Restaurants-g60763CA202ms
01 Live demo
Any URL in. HTML or JSON out.
The Crawling API, typed live. Crawl a page for rendered HTML, or scrape it for structured JSON. Hover to pause and read.
Run your first request in minutes. Up to 20,000 free requests, no credit card.Start free
02 Capabilities
One endpoint, the whole stack underneath.
Every hard part of crawling at scale, handled for you: a real browser, a residential network, and bot defenses cleared on every request.
render
Full JavaScript rendering
A real browser executes the page, so dynamically loaded content, infinite scroll and single-page apps are captured, not just the initial HTML.
proxies
140M residential IPs
Every request rotates a residential IP across 30 geographies, so you reach any site like a real local visitor.
anti-bot
Blocks and CAPTCHAs handled
Bot walls, CAPTCHAs and rate limits are cleared automatically. Nothing to solve, nothing to maintain when a site changes.
extract
Autoparse and scrapers
Get the full rendered HTML, or add autoparse=true or scraper=… to return title, content, prices, images and links as JSON.
capture
Screenshots
Add screenshot=true to capture a full-page image of the rendered page, stored and ready to download.
scale
Async and cloud storage
Run asynchronously with webhooks and the crawler, and keep every crawled page in cloud storage. See the live demo.
03 Parameters
Every request, fully parameterized.
One endpoint, controlled by query parameters. Geotarget, render, parse, screenshot, store and session, all from the same call.
token=PRIVATE_TOKEN
Required. Your API access token.
url=https://crawlbase.com/ip
Required. A URL to crawl.
country=US
Proxy location country, +48 countries.
scraper=scraper-name
Crawl then extract with a scraper.
screenshot=true
Take screenshots of browser as images.
format=html
Response format: html, json or md (Markdown).
store=true
Store your crawled or scraped data in the cloud.
device=mobile
Crawl on mobile devices or desktop.
autoparse=true
Crawling API can autoparse any web page.
tor_network=true
Crawl onion websites over the Tor network.
04 How it works
From URL to data in one call.
Every request moves through the same path. You send a URL, we operate everything in between.
01
You Send the URL
Pass any URL with your token, plus any parameters you need: country, render waits, parsing, screenshots or storage.
02
We Rotate a proxy
A residential IP and geography that reach the site cleanly, drawn from 140M IPs across 30 regions.
03
We Render the page
A real browser loads the page so JavaScript, dynamic content and infinite scroll render before capture.
04
We Clear the anti-bots
Bot checks, CAPTCHAs and rate limits are handled automatically. Nothing to solve, nothing to maintain.
05
We Return HTML or JSON
The fully rendered HTML comes back, typed JSON when you add a scraper or autoparse, clean Markdown when you request it, or an image when you ask for a screenshot.
05 Use cases
What teams build on the Crawling API.
USE / 01Commerce
Price and catalog monitoring
Track prices, stock and listings across retailers and marketplaces, parsed to JSON on every crawl.
USE / 02Search
SERP and SEO tracking
Crawl search results and competitor pages to monitor rankings, snippets and content at scale.
USE / 03AI
Training data and RAG
Feed clean rendered pages and structured JSON into models, RAG pipelines and agents through one API.
USE / 04Growth
Lead and contact discovery
Crawl directories, profiles and listings to build and enrich pipelines.
USE / 05Research
Market and content intelligence
Aggregate news, reviews and public data to inform product, pricing and strategy.
USE / 06Coverage
Any site, one API
Crawl any public URL with the same token, from a single page to millions with the async crawler.
06 Pricing
One rate card, shared with every product.
Crawling API requests spend credits on the platform ladder: rates fall as monthly volume grows, and only successful requests are billed.
Pay as you go$3.00 down to $0.02 per 1,000 successful requests
Only successful requests are billed, and rates fall as volume grows. Optional subscriptions from $99 / mo. Start free with up to 20,000 requests, no credit card.
07 Notes
Good to know.
Test for free
Up to 20,000 requests are free, with no credit card. The same token works across the Crawling API and every scraper.
Simple, usage-based pricing
Pay for what you crawl, no long-term contracts, cancel any time. See the full breakdown on the pricing page.
Fully documented
Every parameter and response is covered in the Crawling API docs, with copy-paste examples for each SDK.
GDPR and CCPA compliant
Crawlbase implements consumer-protection standards globally, with fairness and transparency built into how data is handled.
08 Why Crawlbase
Built to crawl the web at scale.
The Crawling API runs on the same network that serves 46,000+ paying customers and 70,000+ developers. No proxies to buy, no browsers to run, nothing to patch when a site changes.
99%
Average request success rate
46K+
Paying customers on the network
30
Geographies for accurate local results
99.99%
Network uptime
One token, official SDKs for Node, Python, Ruby, PHP, Java, .NET and Go, and a residential network underneath.
09 FAQ
Crawling API questions.
A single REST API that crawls any URL through residential proxies in a real browser, clears bot checks and CAPTCHAs, and returns the fully rendered HTML. Add a scraper or autoparse for structured JSON, or screenshot=true for an image.
Yes. A real browser executes the page, so dynamically loaded content, infinite scroll and single-page apps are captured, not just the initial HTML. Add ajax_wait or page_wait for slow pages.
Yes. Add scraper=generic-extractor for universal JSON, autoparse=true for popular sites like Amazon and Google, or a named scraper for a specific site. Otherwise you get the full rendered HTML to parse yourself.
Crawlbase routes each request through rotating residential IPs across 30 geographies and clears bot checks automatically. You do not manage proxies or solve CAPTCHAs, and there is nothing to maintain when a site changes its setup.
Yes. Add screenshot=true and the API captures a full-page screenshot of the rendered page as an image, stored for you for up to an hour.
Any. The Crawling API is a plain REST endpoint, with official SDKs for Node, Python, Ruby, PHP, Java, .NET and Go, so it drops into your existing stack.
Start free with up to 20,000 requests and no credit card. After that you pay per successful request on one graduated rate card shared by every Crawlbase product, with optional subscriptions from $99 a month. See the pricing page for the full rate card.
Start crawling the web.
Skip the proxies and blocks.
Free to begin with up to 20,000 requests. One token for the Crawling API and every scraper.