Building a Distributed Crawling EngineOrchestrate in Node.js, Execute on Crawlbase

Web crawling stops being a scraping problem long before it becomes a parsing problem. The hard part isn't downloading HTML or extracting data. It's coordinating millions of fetches across thousands of domains without losing work, crawling the same page twice, or spending your time maintaining proxy pools and browser infrastructure instead of building your application.

That's where the architecture changes. A production crawler needs queues, workers, retries, proxy rotation, JavaScript rendering, anti-bot handling, and a durable crawl frontier that can continuously expand without revisiting the same URLs. In short, you're solving a distributed systems problem.

The better approach is to separate crawl orchestration from crawl execution. Your application decides what to crawl, how deep to crawl, and what to extract. The crawling infrastructure handles queueing, concurrency, retries, rendering, and network reliability, leaving your engine focused on business logic instead of infrastructure.

In this article, we'll build a distributed crawling engine in Node.js that follows exactly that design. The engine owns crawl policy, URL deduplication, HTML parsing, result storage, and recursive frontier expansion, while Crawlbase provides the distributed execution layer through the Crawlbase Enterprise Crawler and the Crawlbase Crawling API. The Enterprise Crawler acts as a managed queue and worker fleet that asynchronously delivers crawl results to a webhook, while the Crawling API provides an inline fetch path for latency-sensitive requests. The result is a crawler that scales with Crawlbase's infrastructure while remaining small enough to understand in a few hundred lines of code.

Every implementation shown here comes from the companion GitHub repository. Clone it, follow each implementation stage alongside the article, and you'll end up with a production-ready foundation that you can extend with your own extraction logic, storage backend, and crawl policies.

Designing a distributed crawler

Every distributed crawler continuously solves four problems:

1. Managing the crawl frontier: deciding what to crawl next.
2. Executing crawls: fetching pages reliably at scale.
3. Processing results: extracting and storing useful data.
4. Expanding the crawl: turning newly discovered links into future crawl jobs.

These responsibilities are tightly connected. Every completed crawl produces both data and new URLs. Those URLs pass through the crawl policy, become part of the frontier, and are eventually fetched again, forming a continuous feedback loop.

Crawl frontier

The frontier is the crawler's memory of pending work. Every discovered URL is normalized, checked against the crawl policy, deduplicated, and scheduled for crawling. Without a managed frontier, recursive crawls quickly revisit the same pages or wander outside their intended scope.

Crawl execution

Fetching pages at scale is largely an infrastructure problem. It requires worker coordination, retries, proxy rotation, browser rendering, and anti-bot handling before your parser ever sees the HTML. Building and operating that infrastructure is often more complex than the crawler itself.

Result processing

Completed crawls need to flow into an independent processing pipeline where pages are parsed, structured data is extracted, and results are persisted. Keeping retrieval separate from processing makes both systems easier to maintain and evolve.

Recursive expansion

Every processed page produces new links. After passing depth, domain, and deduplication checks, those links become new crawl jobs and re-enter the frontier. This recursive feedback loop continues until there are no more in-scope pages to visit.

The engine we build owns the frontier, crawl policy, parsing, storage, and recursive expansion. Crawlbase provides the distributed queue, worker fleet, retries, browser rendering, proxy rotation, and anti-bot infrastructure. That separation keeps the engine focused on application logic rather than crawling infrastructure.

What we're building

A small Express service and a set of scripts that together form a self-sustaining crawler:

• A one-time setup script creates a named Crawler queue bound to your webhook.
• A seed script pushes starting URLs into the queue.
• Crawlbase crawls each URL and POSTs the result to your webhook.
• The webhook parses the page, stores a record, discovers links, and pushes the in-scope, not-yet-seen links back into the queue.

The result is a crawler that scales with Crawlbase's concurrency rather than your server's, and that you can reason about in a few hundred lines of code.

Architecture overview

The flow is a loop. The seed script primes the engine, which pushes URLs into the Enterprise Crawler. The Crawler is the distributed piece: it holds the queue, runs the crawls at its configured concurrency through Crawlbase's proxy network, and handles retries and anti-bot resolution against the target websites. When a crawl finishes, Crawlbase POSTs the page back to the engine's webhook. The engine's parser produces a record for the store and a list of outbound links, which flow back through deduplication and depth checks into the next round of pushes.

Crawlbase owns retrieval, concurrency, retries, and proxy/anti-bot infrastructure. The engine owns crawl policy (what is in scope, how deep) and what to do with the data. The hero diagram above also shows the dotted edge: the Crawling API used synchronously for on-demand, single-page fetches when you need a result inline instead of through the queue.

Getting the project ready

Now that we've established the architecture, let's get the project running locally. The companion repository contains the complete implementation used throughout this article. We'll work from the final/ project while referring to the incremental checkpoints under steps/ as we build each component.

You'll need:

• Node.js 18 or newer (the setup script uses the built-in fetch).
• A Crawlbase account to obtain your Request tokens.
• A publicly accessible webhook endpoint. For local development, expose your Express server using Cloudflare Tunnel (cloudflared) or ngrok, then point CALLBACK_URL to https://<your-public-host>/webhook.

Clone the repository and install the project from the final/ directory:

git clone https://github.com/ScraperHub/building-a-distributed-crawling-engine.git
cd building-a-distributed-crawling-engine/final
npm install
cp .env.example .env

Next, edit .env with your Crawlbase token and public callback URL. Throughout the rest of the article, every implementation step maps directly to a checkpoint under steps/, while final/ always contains the complete runnable project. The repository's README includes an article-to-code reference for every section.

Step 1: Centralize configuration

The engine starts with a single source of truth for its configuration. Rather than scattering secrets and crawl-policy values throughout the codebase, everything is loaded from environment variables and validated during startup. If a required value such as the Crawlbase token is missing, the engine fails immediately instead of failing later during a crawl.

The complete implementation lives in final/src/config.js in the companion repository.

const config = {
  crawlbaseToken: required('CRAWLBASE_TOKEN'),
  crawlerName: process.env.CRAWLER_NAME || 'distributed-engine',
  callbackUrl: process.env.CALLBACK_URL || '',
  port: Number(process.env.PORT || 3000),
  maxDepth: Number(process.env.MAX_DEPTH || 2),
  allowedDomains: (process.env.ALLOWED_DOMAINS || '')
    .split(',')
    .map((domain) => domain.trim().toLowerCase())
    .filter(Boolean),
};

Most of these values define the engine's crawl policy rather than its infrastructure. MAX_DEPTH limits how far the crawler can recursively expand, while ALLOWED_DOMAINS keeps the crawl scoped to the sites you explicitly permit. The same CRAWLBASE_TOKEN authenticates both the Enterprise Crawler and the Crawling API, so one credential is all the engine needs to communicate with Crawlbase.

With the configuration in place, the next step is creating the distributed crawler that will execute our crawl jobs.

Step 2: Create the distributed queue

In Crawlbase, the distributed queue is the Enterprise Crawler: a managed queue and worker fleet that accepts URLs, crawls them asynchronously, and delivers the results to either a webhook or Cloud Storage. We'll use webhook delivery by supplying a callback_url when the crawler is created.

The setup logic is implemented in final/src/setup-crawler.js. The Management API expects the Crawlbase token in the URL path rather than the query string, so the script constructs the endpoint accordingly.

const endpoint = `https://api.crawlbase.com/crawler/${config.crawlbaseToken}`;
const response = await fetch(endpoint, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    name: config.crawlerName,
    callback_url: config.callbackUrl,
  }),
});

Create the crawler once:

npm run setup

Once the crawler is bound to your webhook, adding work to the queue becomes a standard Crawling API request with two additional parameters. crawler identifies the target queue, while callback=true tells Crawlbase to process the request asynchronously instead of returning the page immediately. The request returns a request ID (rid) as soon as the URL is accepted into the queue, allowing the actual crawl to happen in the background.

The push logic is implemented in final/src/crawlbase-client.js.

async function pushToCrawler(url, depth) {
  const response = await api.get(url, {
    crawler: config.crawlerName,
    callback: true,
    callbackHeaders: `X-Crawl-Depth|${depth}`,
  });

  // Surface auth/quota errors instead of silently dropping the URL.
  if (response.statusCode !== 200) {
    throw new Error(`Crawler push failed (${response.statusCode}): ${response.body}`);
  }

  // The push response is the small JSON envelope { "rid": "..." }, but it is
  // not served with a JSON content-type, so parse the body ourselves.
  const parsed = response.json || parseRid(response.body);
  return parsed && parsed.rid;
}

Two implementation details are worth noting. First, although the push response contains a JSON payload, it isn't returned with a JSON content type, so the helper parses the response body instead of relying on automatic deserialization. Second, callbackHeaders keeps the engine stateless. The current crawl depth is attached to the request as X-Crawl-Depth, which Crawlbase echoes back when delivering the webhook. That allows the engine to recover the crawl depth without maintaining its own request state.

The same helper is used by the seed script, implemented in final/src/seed.js, to enqueue the initial set of URLs.

for (const url of SEED_URLS) {
  if (!shouldCrawl(url, 0)) {
    console.log(`Skipped (filtered or duplicate): ${url}`);
    continue;
  }
  const rid = await pushToCrawler(url, 0);
  console.log(`Queued ${url} -> rid ${rid}`);
}

Running npm run seed prints one Queued <url> -> rid <rid> line for every accepted seed. At this point, the distributed crawler is running. The URLs now live inside Crawlbase's queue, and the first crawl results will begin arriving at the webhook we'll implement next.

Step 3: Process crawl results with a webhook

Once the crawler starts processing URLs, the engine needs a way to receive the results. Every completed crawl is delivered to the callback_url configured earlier. The HTML page arrives as the request body, while metadata such as pc_status, original_status, rid, url, and any custom callback headers are sent as HTTP headers.

The webhook implementation lives in final/src/server.js. Two details matter here. First, Crawlbase delivers responses gzip-compressed, so the server uses express.raw() to receive the request body as a Buffer, which Express transparently decompresses. Second, webhook acknowledgements should be as fast as possible: acknowledge immediately, process after the response has been sent.

app.use(express.raw({ type: '*/*', limit: '10mb' }));

app.post('/webhook', (req, res) => {
  // The monitoring bot probes this endpoint to detect outages. Acknowledge
  // it as a no-op - it carries no real crawl result to process. Crawlbase
  // expects a 2xx with an empty body, so end the response without content.
  if ((req.headers['user-agent'] || '').includes('Crawlbase Monitoring Bot')) {
    return res.status(200).end();
  }

  const pcStatus = Number(req.headers['pc_status']);
  const rid = req.headers['rid'];
  const url = req.headers['url'];
  const depth = Number(req.headers['x-crawl-depth'] || 0);

  const html = req.body.toString('utf8');
  res.status(200).end();

  if (pcStatus !== 200 || !rid || !url) {
    return;
  }

  setImmediate(() => handleResult({ rid, url, depth, html }));
});

The handler also branches on pc_status rather than the target website's original_status. A pc_status of 200 indicates Crawlbase successfully completed the crawl after applying any required retries or anti-bot handling, so failed requests can simply be ignored by the engine.

Notice that the actual processing happens inside setImmediate(), after the response has already been acknowledged. This keeps the webhook responsive regardless of how much parsing or storage work happens next and allows Crawlbase to continue delivering new crawl results without waiting for downstream processing to finish.

At this stage, the engine is receiving crawl results, but it still isn't doing anything useful with them. The next step is parsing each page, extracting structured data, discovering new links, and feeding them back into the crawler.

Step 4: Extract data and expand the frontier

This is where the crawler becomes self-sustaining. Every page delivered to the webhook produces two outputs: a structured record that can be stored, and a new set of links that may become future crawl jobs. The frontier sits between those two stages, ensuring the crawl remains bounded, deterministic, and free from duplicate work.

The frontier implementation lives in final/src/frontier.js.

function shouldCrawl(rawUrl, depth) {
  if (depth > config.maxDepth) {
    return false;
  }
  const url = normalize(rawUrl);
  if (!url || !isAllowed(url) || seen.has(url)) {
    return false;
  }
  seen.add(url);
  return true;
}

The shouldCrawl helper acts as the engine's gatekeeper. Every discovered URL is normalized, checked against the allowed domains, validated against the maximum crawl depth, and recorded as seen before it's accepted back into the crawl. That single function enforces the crawler's scope, guarantees termination, and prevents duplicate work.

Once a page passes through the webhook, the engine extracts its data, persists the result, and evaluates every outbound link for the next round of crawling. The extraction and storage logic is implemented in final/src/extract.js, while the orchestration happens inside the webhook handler in final/src/server.js.

async function handleResult({ rid, url, depth, html }) {
  const { record, links } = extract(html, url);
  save(rid, { rid, url, depth, crawledAt: new Date().toISOString(), ...record });

  let queued = 0;
  for (const link of links) {
    if (shouldCrawl(link, depth + 1)) {
      try {
        await pushToCrawler(link, depth + 1);
        queued += 1;
      } catch (error) {
        console.error(`Failed to queue ${link}: ${error.message}`);
      }
    }
  }
  console.log(`[${rid}] depth=${depth} url=${url} queued=${queued} new links`);
}

Every outbound link passes through the frontier before it can become another crawl job. Valid URLs are immediately pushed back into the Enterprise Crawler, while duplicates, out-of-scope domains, and pages beyond the configured depth are discarded. The engine never waits for those pages to be crawled; it simply queues them and continues processing the current result. Crawlbase handles the distributed execution, and the webhook receives the next completed page when it's ready.

That recursive feedback loop is the core of the architecture. Each completed crawl generates more work until there are no new in-scope URLs left to discover, allowing the engine to continuously expand the crawl without maintaining its own worker pool or crawl queue.

The complete crawl lifecycle

At this point, every piece of the engine is in place. With your tunnel running and CALLBACK_URL pointing to it, the entire crawl starts with just three commands:

npm run setup   # once: create the crawler and bind it to your webhook
npm start       # start the webhook consumer
npm run seed    # enqueue the initial URLs

From there, the engine runs autonomously. Seed URLs are queued, Crawlbase processes them asynchronously, completed pages are delivered to the webhook, structured data is extracted and stored, and every newly discovered link is evaluated before being pushed back into the queue. The cycle repeats until there are no more in-scope pages left to crawl.

Because the engine only orchestrates the workflow, throughput is determined by Crawlbase's worker concurrency rather than your application's resources. Scaling the crawl doesn't require additional worker processes or changes to the engine itself; it means allowing Crawlbase to process more crawl jobs concurrently. As the crawl progresses, you can monitor activity through the server logs and the growing collection of JSON records under the results/ directory.

The architecture also supports synchronous page retrieval when a queued crawl isn't appropriate. The fetchInline helper in final/src/crawlbase-client.js uses the Crawling API to fetch a single page immediately using the same Crawlbase token. In practice, the Enterprise Crawler is the right choice for high-volume asynchronous crawls, while the Crawling API is better suited for latency-sensitive requests such as user lookups or AI agents retrieving context on demand.

Production considerations

The implementation we've built is intentionally small so the architecture remains easy to understand. Before using it in production, there are a few areas worth strengthening.

• Move the frontier into shared storage. The example stores the seen set in memory, which means deduplication is limited to a single process and disappears on restart. For multiple engine instances or long-running crawls, move the frontier into Redis or another shared datastore keyed by normalized URLs.
• Manage secrets outside the application. The example loads the Crawlbase token from .env for local development. In production, inject credentials through your platform's secret manager instead of storing them in configuration files.
• Keep the webhook fast and reliable. The webhook should acknowledge requests immediately and defer expensive processing until after the response is sent. Slow or unavailable endpoints trigger delivery retries, so protecting the endpoint with a shared secret or custom callback headers helps improve both reliability and security.
• Consider Crawlbase Cloud Storage for asynchronous processing. Webhooks are ideal when you want to process crawl results as soon as they're available, but they aren't the only delivery option. When the Enterprise Crawler is configured to use Cloud Storage, every completed crawl is automatically persisted without requiring a webhook, and your application retrieves results on its own schedule through the Storage API. This works well for batch processing, environments where exposing an HTTPS endpoint isn't practical, or workflows that benefit from a permanent URL for each crawled page. See the documentation on Storage delivery mode for details.
• Monitor queue growth. Enterprise Crawlers enforce concurrency, queue capacity, and push-rate limits to keep crawls predictable. If your producers outpace your consumers, monitor queue size and pause or purge runaway crawls through the Management API before they consume unnecessary resources. As your workload grows, you can request higher concurrency or push-rate limits from the Crawlbase Support Dashboard.
• Choose the right rendering mode. Start with a Normal token whenever possible. If a target relies heavily on client-side rendering or returns challenge pages, switch to the JavaScript Crawler and use rendering parameters such as page_wait or ajax_wait to wait for dynamic content before extraction.
• Respect crawl boundaries. Keep the crawler within domains you own or are authorized to access, follow each site's robots.txt and terms of service where applicable, and configure sensible crawl limits to avoid generating unnecessary traffic.

Conclusion

We've built a distributed crawling engine by separating crawl orchestration from crawl execution. The engine owns crawl policy, URL deduplication, parsing, storage, and recursive frontier expansion, while Crawlbase provides the distributed queue, worker fleet, retries, browser rendering, and proxy infrastructure. That separation keeps the application small, easy to reason about, and focused on the logic that makes it unique.

The example implementation intentionally favors simplicity over production readiness. As your workloads grow, you can replace the in-memory frontier with Redis, persist crawl results to a database, or run multiple stateless engine instances behind the same distributed queue without changing the overall architecture.

Frequently asked questions

When should I use the Enterprise Crawler instead of the Crawling API?

The two services solve different problems. Use the Enterprise Crawler for asynchronous, large-scale crawls where you need queueing, retries, concurrency, and webhook or Cloud Storage delivery. Use the Crawling API when you need a page immediately, such as powering a user-facing request or giving an AI agent real-time context. A good rule of thumb is to use the queue for throughput and the Crawling API for low-latency requests.

Do I need to expose a public webhook?

No. This article uses a webhook because it's the most responsive way to process crawl results as they become available. Alternatively, you can configure an Enterprise Crawler to deliver results to Crawlbase Cloud Storage and retrieve them later through the Storage API. That's often a better fit for batch processing workflows or environments where exposing a public HTTPS endpoint isn't practical.

When should I use a JavaScript token instead of a Normal token?

Start with a Normal token whenever possible because it's faster and more cost-effective. If the target website renders content client-side, returns an empty HTML shell, or presents anti-bot challenges that require a browser, switch to a JavaScript request and use rendering parameters such as page_wait or ajax_wait when needed.

How can I make the crawler process URLs faster?

The engine itself is intentionally lightweight; it only orchestrates the crawl. Overall throughput is determined by the Enterprise Crawler's concurrency rather than your application's resources. If your workload requires higher throughput than your current limits allow, you can request an increase through the Support Dashboard.

Can I run multiple instances of this engine?

Yes. The architecture is designed for horizontal scaling. The only component that needs to change is the in-memory frontier. Replacing the local seen set with shared storage such as Redis allows multiple webhook consumers to coordinate crawl state while sharing the same Enterprise Crawler queue.

NZ

Neil Zamora

Senior Architect · Crawlbase

Senior architect at Crawlbase, focused on the systems behind large-scale crawling: proxy rotation, anti-bot resilience, and the APIs that hide that complexity.

IK

Ian Kalvin

Technical Support Engineer · Crawlbase

Technical support engineer at Crawlbase, writing from the front line of what actually breaks in production scraping and proxy setups.