Product / Crawling APICrawling API.
Crawling API.
Any URL, fully rendered.
Send any URL and get the fully rendered HTML back, through 140M residential IPs with anti-bot handling built in.
Add a scraper or autoparse for structured JSON, or a screenshot.
Live crawl feed1.24M req/minStreaming
200target.com/p/-/A-79404211JP201ms
200zillow.com/homes/for_sale/BR212ms
200indeed.com/jobs?q=developerNL160ms
200booking.com/searchresults.html?ss=ParisGB128ms
200ebay.com/itm/204512389011DE49ms
200github.com/crawlbaseSG204ms
200reddit.com/r/programmingCA208ms
200glassdoor.com/Reviews/index.htmFR79ms
200stackoverflow.com/questions/11227809IN190ms
200reddit.com/r/programmingSG182ms
200zillow.com/homes/for_sale/NL172ms
200linkedin.com/jobs/searchSG215ms
200amazon.com/dp/B08N5WRWNWGB203ms
200producthunt.com/posts/notionBR170ms
200stackoverflow.com/questions/11227809IN91ms
200google.com/search?q=web+scrapingJP201ms
200yelp.com/biz/blue-bottle-coffeeGB40ms
200tripadvisor.com/Restaurants-g60763DE193ms
200linkedin.com/jobs/searchBR71ms
200booking.com/searchresults.html?ss=ParisDE176ms
200ebay.com/itm/204512389011DE143ms
200zillow.com/homes/for_sale/JP50ms
200producthunt.com/posts/notionIN211ms
200stackoverflow.com/questions/11227809SG185ms
200producthunt.com/posts/notionIN184ms
200producthunt.com/posts/notionGB119ms
200target.com/p/-/A-79404211JP201ms
200zillow.com/homes/for_sale/BR212ms
200indeed.com/jobs?q=developerNL160ms
200booking.com/searchresults.html?ss=ParisGB128ms
200ebay.com/itm/204512389011DE49ms
200github.com/crawlbaseSG204ms
200reddit.com/r/programmingCA208ms
200glassdoor.com/Reviews/index.htmFR79ms
200stackoverflow.com/questions/11227809IN190ms
200reddit.com/r/programmingSG182ms
200zillow.com/homes/for_sale/NL172ms
200linkedin.com/jobs/searchSG215ms
200amazon.com/dp/B08N5WRWNWGB203ms
200producthunt.com/posts/notionBR170ms
200stackoverflow.com/questions/11227809IN91ms
200google.com/search?q=web+scrapingJP201ms
200yelp.com/biz/blue-bottle-coffeeGB40ms
200tripadvisor.com/Restaurants-g60763DE193ms
200linkedin.com/jobs/searchBR71ms
200booking.com/searchresults.html?ss=ParisDE176ms
200ebay.com/itm/204512389011DE143ms
200zillow.com/homes/for_sale/JP50ms
200producthunt.com/posts/notionIN211ms
200stackoverflow.com/questions/11227809SG185ms
200producthunt.com/posts/notionIN184ms
200producthunt.com/posts/notionGB119ms
01 Live demo
Any URL in. HTML or JSON out.
The Crawling API, typed live. Crawl a page for rendered HTML, or scrape it for structured JSON. Hover to pause and read.
Run your first request in minutes. 1,000 free requests, no credit card.Start free
02 Capabilities
One endpoint, the whole stack underneath.
Every hard part of crawling at scale, handled for you: a real browser, a residential network, and bot defenses cleared on every request.
render
Full JavaScript rendering
A real browser executes the page, so dynamically loaded content, infinite scroll and single-page apps are captured, not just the initial HTML.
proxies
140M residential IPs
Every request rotates a residential IP across 30 geographies, so you reach any site like a real local visitor.
anti-bot
Blocks and CAPTCHAs handled
Bot walls, CAPTCHAs and rate limits are cleared automatically. Nothing to solve, nothing to maintain when a site changes.
extract
Autoparse and scrapers
Get the full rendered HTML, or add autoparse=true or scraper=… to return title, content, prices, images and links as JSON.
capture
Screenshots
Add screenshot=true to capture a full-page image of the rendered page, stored and ready to download.
scale
Async and cloud storage
Run asynchronously with webhooks and the crawler, and keep every crawled page in cloud storage. See the live demo.
03 Parameters
Every request, fully parameterized.
One endpoint, controlled by query parameters. Geotarget, render, parse, screenshot, store and session, all from the same call.
token=PRIVATE_TOKEN
Required. Your API access token.
url=https://crawlbase.com/ip
Required. A URL to crawl.
country=US
Proxy location country, +48 countries.
scraper=scraper-name
Crawl then extract with a scraper.
screenshot=true
Take screenshots of browser as images.
format=html
Response format: html, json or md (Markdown).
store=true
Store your crawled or scraped data in the cloud.
device=mobile
Crawl on mobile devices or desktop.
autoparse=true
Crawling API can autoparse any web page.
tor_network=true
Crawl onion websites over the Tor network.
04 How it works
From URL to data in one call.
Every request moves through the same path. You send a URL, we operate everything in between.
01
You Send the URL
Pass any URL with your token, plus any parameters you need: country, render waits, parsing, screenshots or storage.
02
We Rotate a proxy
A residential IP and geography that reach the site cleanly, drawn from 140M IPs across 30 regions.
03
We Render the page
A real browser loads the page so JavaScript, dynamic content and infinite scroll render before capture.
04
We Clear the anti-bots
Bot checks, CAPTCHAs and rate limits are handled automatically. Nothing to solve, nothing to maintain.
05
We Return HTML or JSON
The fully rendered HTML comes back, typed JSON when you add a scraper or autoparse, clean Markdown when you request it, or an image when you ask for a screenshot.
05 Use cases
What teams build on the Crawling API.
USE / 01Commerce
Price and catalog monitoring
Track prices, stock and listings across retailers and marketplaces, parsed to JSON on every crawl.
USE / 02Search
SERP and SEO tracking
Crawl search results and competitor pages to monitor rankings, snippets and content at scale.
USE / 03AI
Training data and RAG
Feed clean rendered pages and structured JSON into models, RAG pipelines and agents through one API.
USE / 04Growth
Lead and contact discovery
Crawl directories, profiles and listings to build and enrich pipelines.
USE / 05Research
Market and content intelligence
Aggregate news, reviews and public data to inform product, pricing and strategy.
USE / 06Coverage
Any site, one API
Crawl any public URL with the same token, from a single page to millions with the async crawler.
06 Pricing
Add the sites you crawl, see the price.
Add the sites you crawl with their monthly volume and request type. We group them by difficulty and type, then price each group on its combined volume, so the more you crawl, the cheaper it gets.
100k / mo
No sites yet. Add one above to start your estimate.
Estimated monthly cost
$0/ mo
≈ $0.00 blended per 1,000 requests
First 1,000 requests free. No credit card.
Start freeCrawling over 1B a month? Talk to us →07 Notes
Good to know.
Test for free
Your first 1,000 requests are free, with no credit card. The same token works across the Crawling API and every scraper.
Simple, usage-based pricing
Pay for what you crawl, no long-term contracts, cancel any time. See the full breakdown on the pricing page.
Fully documented
Every parameter and response is covered in the Crawling API docs, with copy-paste examples for each SDK.
GDPR and CCPA compliant
Crawlbase implements consumer-protection standards globally, with fairness and transparency built into how data is handled.
08 Why Crawlbase
Built to crawl the web at scale.
The Crawling API runs on the same network that serves 46,000+ paying customers and 70,000+ developers. No proxies to buy, no browsers to run, nothing to patch when a site changes.
99%
Average request success rate
46K+
Paying customers on the network
30
Geographies for accurate local results
99.99%
Network uptime
One token, official SDKs for Node, Python, Ruby, PHP, Java, .NET and Go, and a residential network underneath.
09 FAQ
Crawling API questions.
A single REST API that crawls any URL through residential proxies in a real browser, clears bot checks and CAPTCHAs, and returns the fully rendered HTML. Add a scraper or autoparse for structured JSON, or screenshot=true for an image.
Yes. A real browser executes the page, so dynamically loaded content, infinite scroll and single-page apps are captured, not just the initial HTML. Add ajax_wait or page_wait for slow pages.
Yes. Add scraper=generic-extractor for universal JSON, autoparse=true for popular sites like Amazon and Google, or a named scraper for a specific site. Otherwise you get the full rendered HTML to parse yourself.
Crawlbase routes each request through rotating residential IPs across 30 geographies and clears bot checks automatically. You do not manage proxies or solve CAPTCHAs, and there is nothing to maintain when a site changes its setup.
Yes. Add screenshot=true and the API captures a full-page screenshot of the rendered page as an image, stored for you for up to an hour.
Any. The Crawling API is a plain REST endpoint, with official SDKs for Node, Python, Ruby, PHP, Java, .NET and Go, so it drops into your existing stack.
Start free with 1,000 requests and no credit card. Paid plans scale with usage, and the same token works across the Crawling API and every Crawlbase scraper.
Start crawling the web.
Skip the proxies and blocks.
Free to begin with 1,000 requests. One token for the Crawling API and every scraper.