How do I receive the data?

Two ways. Point the crawler at your webhook endpoint and Crawlbase posts each result to it, or store results in Crawlbase cloud storage and pull them when you are ready. Webhook URLs are monitored so delivery stays reliable.

Can I monitor and control my crawls?

Yes. The dashboard shows live stats for each crawler, and the Crawler API lets you check stats and manage crawls programmatically. You can pause and resume crawling to match your budget and needs.

Product / Crawler

Crawler.
Push URLs, pull data, async at scale.

Push millions of URLs to a managed push/pull queue built on the Crawling API, and receive the rendered data on your webhook or in cloud storage.
No queues, retries or proxies to run.

Start free Read the docs

99% success rateAsync push and pullWebhook or cloud storage

Live crawler queue1.24M req/minStreaming

200amazon.com/s?k=laptops&page=3DE79ms

200amazon.com/s?k=laptops&page=1IN131ms

200target.com/c/electronics/-/N-5xtg6JP44ms

200indeed.com/jobs?q=engineer&start=20AU82ms

200crunchbase.com/discover/organization.companiesFR43ms

200aliexpress.com/category/100003070/men.htmlAU195ms

200booking.com/searchresults.html?offset=25DE139ms

200aliexpress.com/category/100003070/men.htmlGB40ms

200aliexpress.com/category/100003070/men.htmlBR170ms

200amazon.com/s?k=laptops&page=3AU154ms

200target.com/c/electronics/-/N-5xtg6NL66ms

200target.com/c/electronics/-/N-5xtg6AU203ms

200booking.com/searchresults.html?offset=25JP87ms

200yelp.com/search?find_desc=cafe&start=30US199ms

200glassdoor.com/Reviews/company-reviews.htmNL107ms

200zillow.com/homes/for_sale/2_p/DE50ms

200target.com/c/electronics/-/N-5xtg6GB169ms

200amazon.com/s?k=laptops&page=2DE57ms

200walmart.com/browse/electronics/3944US144ms

200ebay.com/sch/i.html?_nkw=gpu&_pgn=1JP47ms

200amazon.com/s?k=laptops&page=1US218ms

200tripadvisor.com/Hotels-g60763-oa30NL107ms

200crunchbase.com/discover/organization.companiesGB184ms

200aliexpress.com/category/100003070/men.htmlES159ms

200indeed.com/jobs?q=engineer&start=20DE164ms

200yelp.com/search?find_desc=cafe&start=30DE64ms

200amazon.com/s?k=laptops&page=3DE79ms

200amazon.com/s?k=laptops&page=1IN131ms

200target.com/c/electronics/-/N-5xtg6JP44ms

200indeed.com/jobs?q=engineer&start=20AU82ms

200crunchbase.com/discover/organization.companiesFR43ms

200aliexpress.com/category/100003070/men.htmlAU195ms

200booking.com/searchresults.html?offset=25DE139ms

200aliexpress.com/category/100003070/men.htmlGB40ms

200aliexpress.com/category/100003070/men.htmlBR170ms

200amazon.com/s?k=laptops&page=3AU154ms

200target.com/c/electronics/-/N-5xtg6NL66ms

200target.com/c/electronics/-/N-5xtg6AU203ms

200booking.com/searchresults.html?offset=25JP87ms

200yelp.com/search?find_desc=cafe&start=30US199ms

200glassdoor.com/Reviews/company-reviews.htmNL107ms

200zillow.com/homes/for_sale/2_p/DE50ms

200target.com/c/electronics/-/N-5xtg6GB169ms

200amazon.com/s?k=laptops&page=2DE57ms

200walmart.com/browse/electronics/3944US144ms

200ebay.com/sch/i.html?_nkw=gpu&_pgn=1JP47ms

200amazon.com/s?k=laptops&page=1US218ms

200tripadvisor.com/Hotels-g60763-oa30NL107ms

200crunchbase.com/discover/organization.companiesGB184ms

200aliexpress.com/category/100003070/men.htmlES159ms

200indeed.com/jobs?q=engineer&start=20DE164ms

200yelp.com/search?find_desc=cafe&start=30DE64ms

01 Live demo

Push a URL. Get it delivered.

The Crawler, typed live. Push a URL to the queue, then receive the rendered result on your webhook. Hover to pause and read.

ready

keys 1-2 switch · click to pauserun your own URL

Run your first request in minutes. 1,000 free requests, no credit card.Start free

02 Capabilities

Crawling at scale, queues included.

Everything that makes large crawls hard, run for you: an async queue, retries, delivery and monitoring, all on top of the Crawling API.

async

Asynchronous push and pull

Push as many URLs as you like and keep going. Crawlbase queues, schedules and renders them in the background, so your client never blocks.

built-on

Built on the Crawling API

Every Crawling API feature is kept: JavaScript rendering, residential proxies, geotargeting, parameters and anti-bot handling on each request.

deliver

Webhook delivery

Point the crawler at your endpoint and each result is posted to it. Crawlbase monitors your webhook so delivery stays accurate and reliable.

storage

Cloud storage

Prefer to pull? Keep every crawled page in Crawlbase cloud storage and fetch it on your own schedule. See Cloud Storage.

monitor

Custom crawlers, live stats

Name a crawler per workload and watch it in real time. Check stats through the API, and pause or resume to match your budget.

fresh

Fresh data, fewer retries

Every page is crawled live, nothing cached. The push/pull system pushes success rates close to 100%, so client-side retries all but disappear.

03 How it works

Migrate in two extra parameters.

Keep your Crawling API calls. Add a callback and a crawler name, and you are async.

Create a crawler

Open the Crawler dashboard, create a named crawler and point it at your webhook or cloud storage.

Push the URLs

Call the Crawling API with callback=true and crawler=YourCrawlerName, for one URL or millions.

We queue and render

Crawlbase schedules each request, rotates a residential proxy, renders the page and retries any failures.

Deliver the result

Each rendered page is posted to your webhook or written to cloud storage, as HTML or structured JSON.

Pull and monitor

Pull from storage when ready and track every crawler live, with pause and resume on demand.

04 Use cases

What teams build on the Crawler.

USE / 01Scale

Millions of pages

Push entire catalogs or sitemaps and let the queue work them through, with no client-side scheduling.

USE / 02Pipelines

Data into your stack

Deliver rendered pages straight to a webhook or storage, ready for your warehouse, index or model.

USE / 03Commerce

Continuous monitoring

Re-crawl prices, stock and listings on a schedule, with fresh data on every pass.

USE / 04AI

Training and RAG corpora

Build large, clean page sets for training and retrieval, pulled from storage in bulk.

USE / 05Migration

Move off your own crawler

Swap your push/pull system for ours with two parameters and drop the proxies, queues and retries.

USE / 06Coverage

Millions of sites

Crawl across millions of supported sites with one crawler and one token.

05 Pricing

Add the sites you crawl, see the price.

Add the sites you crawl with their monthly volume and request type. We group them by difficulty and type, then price each group on its combined volume, so the more you crawl, the cheaper it gets.

100k / mo

No sites yet. Add one above to start your estimate.

Estimated monthly cost

$0/ mo

≈ $0.00 blended per 1,000 requests

First 1,000 requests free. No credit card.

Start free Crawling over 1B a month? Talk to us →

06 Notes

Good to know.

Test for free

Your first 1,000 requests are free, with no credit card. The same token works across the Crawler, the Crawling API and every scraper.

Usage-based pricing

Pay for what you crawl, no long-term contracts, cancel any time. Pause and resume to match your budget. See the full breakdown on the pricing page.

Fully documented

Creating crawlers, callbacks and delivery are all covered in the Crawler docs, with copy-paste examples.

GDPR and CCPA compliant

Crawlbase applies consumer-protection standards globally, with fairness and transparency built into how data is handled.

07 Why Crawlbase

Built to crawl the web at scale.

The Crawler runs on the same network that serves 70,000+ developers and the world's most demanding crawling workloads. No queues to run, no proxies to buy, nothing to patch when a site changes.

99%

Average request success rate

70K+

Customers on the network

Async

Push and pull, queues handled

99.99%

Network uptime

One token across the Crawler, the Crawling API and every scraper, with delivery to your webhook or storage.

08 FAQ

Crawler questions.

An asynchronous crawler built on the Crawling API. You push URLs to a managed push/pull queue and Crawlbase renders each page, retries failures and delivers the result to your webhook or to cloud storage, so you never manage queues, retries or proxies.

The Crawling API is synchronous, you get the response on the same call. The Crawler is asynchronous, you push URLs and the data is delivered later to your webhook or storage. It keeps every Crawling API feature, including JavaScript rendering and residential proxies.

Create a named crawler in the dashboard, then add two parameters to your Crawling API call, callback=true and crawler=YourCrawlerName. That is the only change needed to switch synchronous calls to the async push/pull system.

Two ways. Point the crawler at your webhook endpoint and Crawlbase posts each result to it, or store results in cloud storage and pull them when you are ready. Webhook URLs are monitored so delivery stays reliable.

Yes. The dashboard shows live stats for each crawler, and the Crawler API lets you check stats and manage crawls. You can pause and resume crawling to match your budget and needs.

Yes. Every page is crawled live from the internet at request time. Nothing is cached or served from an earlier crawl, so the data you receive is current.

Crawl the web asynchronously.
Push URLs, we deliver the data.

Free to begin with 1,000 requests. One token for the Crawler, the Crawling API and every scraper.