How to Use the Crawlbase Crawlerasync scraping at scale

When you scrape a few hundred pages, a synchronous loop is fine: send a request, wait for the HTML, parse it, repeat. That model breaks down once you need thousands or millions of pages, because every request blocks your code while a remote browser renders a page and a proxy works through retries. The Crawlbase Crawler exists to remove that wait. You push URLs to it, it queues and crawls them in the background, and it delivers each finished result to a webhook you control.

This tutorial shows you how to use the Crawler end to end in Python: stand up a webhook endpoint that receives results, create a Crawler that points at it, push a batch of URLs through the Crawling API, and read back the crawled HTML as it arrives. Along the way you will see the features that make the Crawler suited to scale: asynchronous processing, normal versus JavaScript requests, paying only for successful requests, and automatic retries. The example targets are neutral public pages, and there is a short note on scraping responsibly near the end.

What you will build

A small but complete asynchronous pipeline. One side is a webhook server that accepts POST callbacks from the Crawler and stores each result. The other side is a push script that submits URLs to your named Crawler through the Crawling API. The pieces you will end up with:

• Webhook receiver. A Flask endpoint that accepts the Crawler's POST callback, decompresses the gzip body, and saves the crawled HTML to disk.
• Crawler definition. A named Crawler in your dashboard, either a normal (TCP) Crawler for static pages or a JavaScript Crawler for client-rendered ones, pointed at your webhook URL.
• Push script. A Python script that sends a list of URLs to the Crawler with callback=true and crawler=YourCrawlerName, and prints the request id (RID) returned for each.
• Result handling. The fields that arrive on each callback, including the RID, the original URL, the status codes, and the page body.

How the async Crawler differs from a direct request

A direct Crawling API call is synchronous: you call api.get(url), your code blocks until the page is fetched and rendered, and you get the HTML back in the same response. That is the right tool for a handful of pages or for interactive work where you want the result immediately.

The Crawler inverts that flow. You push a URL, get back a short request id straight away, and carry on pushing the rest of your batch. The Crawler crawls each page in the background, handling rendering, IP rotation, and retries on its own schedule, and posts the finished result to your webhook when it is ready. Your code never holds a connection open for the duration of a crawl. That decoupling is what lets one batch span thousands of URLs without your process sitting idle, and it is why the async product page recommends it for large jobs. The trade-off: you need a publicly reachable endpoint to receive the callbacks, which is what you build first.

Prerequisites

A few things to have in place. None take long.

Basic Python. You should be comfortable running a script and installing packages with pip. If you are new to handling HTML once it arrives, the BeautifulSoup guide pairs well with this one.

Python 3.8 or later. Confirm with python --version. If you do not have it, install it from python.org and make sure it is on your PATH.

A Crawlbase account and token. Sign up, open your dashboard, and copy your token from the account docs page. You get a normal token and a JavaScript token; use the one that matches the Crawler type you create. Crawlbase includes 1,000 free requests to start, which is enough to work through this guide. Treat the token like a password and keep it out of version control.

A way to expose localhost. The Crawler delivers results over the public internet, so your webhook must be reachable from outside your machine. For local development a tunnelling tool such as ngrok forwards a public URL to your local port. In production you would host the endpoint on a real server.

Set up the project

Create a virtual environment so dependencies stay isolated, then install the two libraries you need: Flask for the webhook server and the official Crawlbase client for the push script.

python --version

python -m venv crawler_env
source crawler_env/bin/activate

pip install crawlbase flask

On Windows, activate the environment with crawler_env\Scripts\activate instead of the source line. The crawlbase package is the official client for pushing URLs through the Crawling API, and flask gives you a minimal web server for the callback endpoint. The gzip module used to decompress the callback body ships with the standard library, so there is nothing more to install.

Step 1: Build the webhook receiver

The Crawler delivers every result as a POST to your callback URL. To be a valid endpoint, your webhook needs to do three things: be reachable from the public internet, accept POST calls, and respond quickly with a 200, 201, or 204 status and no body. The Crawler sends the page body gzip-compressed, so you decompress it before saving. Here is a complete Flask receiver.

# webhook.py
import gzip
from flask import Flask, request, Response

app = Flask(__name__)

@app.route("/webhook/crawlbase", methods=["POST"])
def webhook():
    rid = request.headers.get("rid")
    url = request.headers.get("url")
    pc_status = request.headers.get("pc_status")

    try:
        body = gzip.decompress(request.data).decode("latin1")
    except OSError:
        body = request.data.decode("latin1", errors="replace")

    with open(f"result_{rid}.html", "w", encoding="latin1") as f:
        f.write(body)

    print(f"Received {rid} for {url} (status {pc_status})")
    return Response(status=204)

if __name__ == "__main__":
    app.run(port=8000)

The endpoint reads the RID, the crawled URL, and the Crawlbase status from the request headers, decompresses the gzip body into HTML, and writes it to a file named after the RID so each result lands separately. It returns 204 No Content, exactly what the Crawler wants: a fast, empty acknowledgement. Run it with python webhook.py and the server listens on port 8000.

Now make it public. With the server running, start a tunnel on the same port:

ngrok http 8000

ngrok prints a public forwarding URL such as https://abc123.ngrok-free.app. Your full callback URL is that host plus the route, for example https://abc123.ngrok-free.app/webhook/crawlbase. Keep this handy; you need it in the next step. With ngrok's free tier the URL changes each time you restart it, so re-read it after any restart.

Step 2: Create the Crawler

With a public callback URL in hand, create the Crawler from your dashboard. Open the Crawler section and choose Create new Crawler. You give it three things:

• A name. A unique identifier you will reference when pushing URLs, for example test-crawler.
• A type. Normal (TCP) for static pages or JavaScript for client-rendered pages, as covered above.
• A callback URL. The public webhook URL from Step 1, including the route: https://abc123.ngrok-free.app/webhook/crawlbase.

If you would rather not run your own endpoint, Crawlbase Cloud Storage can act as the callback target and hold results for you to fetch later. For this tutorial we use the webhook you built, since it shows the full callback flow. Once the Crawler is saved, it is ready to accept pushed URLs.

Step 3: Push URLs to the Crawler

Pushing is where the Crawling API comes in. You call it the same way you would for a synchronous crawl, but you add two options: callback=true to tell it this is an asynchronous request, and crawler=test-crawler to name the Crawler that should handle it. Each push returns a request id rather than the page itself. Here is the push script.

# push.py
from crawlbase import CrawlingAPI

api = CrawlingAPI({"token": "YOUR_CRAWLBASE_TOKEN"})

urls = [
    "https://httpbin.org/html",
    "https://example.com/",
    "https://books.toscrape.com/",
]

OPTIONS = {"callback": "true", "crawler": "test-crawler"}

for url in urls:
    response = api.get(url, OPTIONS)
    print(response["body"])

Each call returns immediately with a small JSON body containing the RID, and the Crawler queues the URL for background crawling. By default you can push up to 30 URLs per second; if you need more, Crawlbase support can raise the limit. Note that a normal token pairs with a normal Crawler and a JavaScript token with a JavaScript Crawler, so use the token that matches the type you created.

What the push returns

Run python push.py and you get one RID per URL, in the order you pushed them:

{"rid": "d756c32b0999b1c0507e364f"}
{"rid": "455ee207f6907fbd6168ac1e"}
{"rid": "e9eb6ce579dec207e8973615"}

The RID is your handle on each request. You can use it to look a request up later through the Crawler's management endpoints, and it arrives back on the callback so you can match each result to the URL you pushed. Because the push is asynchronous, the whole batch returns in well under a second; the actual crawling happens after, in the background.

Step 4: Receive results at the webhook

Once a page is crawled, the Crawler POSTs the result to your callback URL. The body is the gzip-compressed page, and the metadata travels in the headers. The default response is HTML, with headers shaped like this:

Content-Type:     text/plain
Content-Encoding: gzip
Original-Status:  200
PC-Status:        200
rid:              the RID you received in the push call
url:              the URL which was crawled

Body: the gzip-compressed HTML of the page

Original-Status is the status the target site returned, and PC-Status is Crawlbase's own status for the crawl, so you can tell a successful fetch from a failed one and react accordingly. If you prefer structured output over raw HTML, pass format=json when pushing, and the body arrives as a JSON object instead:

{
  "pc_status": 200,
  "original_status": 200,
  "rid": "the RID you received in the push call",
  "url": "the URL which was crawled",
  "body": "the HTML of the page"
}

Because we pushed three URLs, the webhook receives three POSTs, each writing its own result_<rid>.html file. With the HTML on disk you can parse it however you like; this is the point where you would load it into BeautifulSoup and pull out the fields you care about, exactly as you would after a synchronous crawl.

Key features that matter at scale

The four-step flow above is the whole pattern. What makes it hold up over millions of URLs is a handful of Crawler behaviours worth calling out.

• Asynchronous processing. Pushing returns a RID instantly and the crawl runs in the background, so your code is never blocked waiting on a slow page. A single process can push a very large batch and then simply receive results as they land.
• Normal versus JavaScript requests. A normal Crawler fetches static HTML cheaply; a JavaScript Crawler renders the page in a browser for client-side content. You choose per Crawler, and JavaScript requests draw more credits than normal ones, so you pay for rendering only when you need it.
• Pay per successful request. You are charged for requests that succeed, not for every attempt, which keeps the cost of a large crawl tied to results rather than effort.
• Automatic retries. If the Crawler delivers to your webhook but your server does not return a successful status, it retries the crawl and redelivers. Those retries do count as successful requests once they land, so keep your endpoint fast and returning 204.

One operational detail: if your webhook goes offline, Crawlbase's monitoring detects it and pauses the Crawler, then resumes once the endpoint is reachable. The combined waiting queues across your Crawlers are capped; a push pauses with an email notice if you hit the ceiling and resumes as the queue drains. You rarely touch any of this directly, but it is why a long-running async job does not silently lose results.

Scaling the pipeline

For a production run the shape stays the same; you change the inputs and the receiving side. A few habits keep large jobs healthy:

• Batch your pushes. Read URLs from a file or a queue and push them in a loop, staying within the 30-per-second default unless you have asked for a higher limit. Store each returned RID so you can reconcile results later.
• Make the webhook durable. Acknowledge fast with a 204, then process the body out of band (write to storage or hand it to a worker queue) rather than parsing inside the request. A slow webhook triggers retries you pay for.
• Watch the status codes. Track PC-Status and Original-Status on each callback so you can tell genuine page failures from transient ones, and feed any that need it back into the queue.

If you would rather not manage callback storage at all, point the Crawler at Crawlbase Cloud Storage and fetch results on your own schedule. For a fuller treatment of standing up a callback service and persisting results, see extracting data with the Crawlbase Crawler and the guide to building a scalable web data pipeline. If your targets are heavily client-rendered, the crawling JavaScript websites walkthrough covers the rendering side in more depth.

Scraping responsibly

The Crawler makes large-scale collection straightforward, which makes responsible use a matter of discipline rather than capability. Scrape only public data, the pages anyone can reach without an account, and stay off anything behind a login or paywall. Check each target site's terms of service and its robots.txt, and treat both as the boundary for what you collect.

Keep your request rate reasonable. The async model can push hard, so set volumes that do not strain the sites you crawl. When the data you collect includes anything tied to identifiable individuals, treat it as personal data and handle it in line with regulations such as GDPR and CCPA: minimise what you keep, aggregate where you can, and do not build profiles of people. The neutral example URLs in this guide (httpbin, example.com, and a sandbox bookstore) exist precisely so you can test the flow without pointing it at anyone's production site.

Frequently Asked Questions (FAQs)

How is the Crawler different from a normal Crawling API call?

A normal Crawling API call is synchronous: you wait for the response and get the HTML back in the same request. The Crawler is asynchronous: you push a URL, get a request id immediately, and the finished result is posted to your webhook later. Use a direct call for a few pages or interactive work, and the Crawler when you need to process thousands or millions of URLs without blocking your code.

What does my webhook have to do to be valid?

It must be reachable from the public internet, accept POST requests, and respond quickly with a 200, 201, or 204 status and no body. The Crawler sends the page gzip-compressed, so decompress the body before using it. Acknowledge fast and do heavier processing afterward, because a slow webhook can trigger retries.

Do I have to use Python?

No. Python is convenient here because of the official Crawlbase client and Flask, but the Crawler is language-agnostic. The push side is a Crawling API call with callback=true and crawler=YourCrawlerName, and the webhook is any HTTP endpoint that accepts a POST. You can build both sides in JavaScript, Ruby, Go, or anything that speaks HTTP.

What is the RID and how do I use it?

The RID (request id) is returned when you push a URL and is echoed back on the callback. It lets you match each incoming result to the URL you submitted, and you can use it to look a request up through the Crawler's management endpoints. Storing the RID for every push is the simplest way to reconcile a large batch as results arrive.

When should I use a JavaScript Crawler instead of a normal one?

Use a JavaScript Crawler when the content you need is rendered client-side, for example a React or Angular app, or a page that fills in after load. A normal (TCP) Crawler is enough for server-rendered, static HTML and costs fewer credits per request. Match the Crawler type to your target, and use the token (normal or JavaScript) that corresponds to it.

How does billing work with retries?

You pay only for successful requests rather than every attempt, and JavaScript requests draw more credits than normal ones. If the Crawler tries to deliver a result but your webhook does not return a success status, it retries and redelivers; those retries count as successful requests once they land. Keeping your endpoint fast and returning 204 avoids paying for avoidable retries.

HR

Hassan Rehan

Software Engineer · Crawlbase

Software engineer at Crawlbase writing hands-on guides on rotating proxies, scraping, and the practical details of wiring proxies into real code.