Rate Limits · Crawlbase Documentation

Default limits

Every account starts with the same generous defaults. You don't need to ask for these — they're already on every token the moment you sign up.

Limit	Default	Scope
Concurrent requests	`20`	per token
Requests per second	~ `20` (derived)	per token
Total monthly requests	up to `51,000,000`	per token
Single-request timeout	`90` seconds	per request
Crawler queue size	`100,000` URLs	per crawler

"Concurrent" means in-flight at the same moment. If each crawl takes 2 seconds and you keep 20 parallel, you'll do roughly 10 req/sec sustained — that math works out to ~864K requests per day per token.

Concurrency, not RPS

Don't try to throttle to "X requests per second" — just cap your worker pool at 20 and let request latency dictate throughput. It's simpler and matches how the limit actually works server-side.

What happens when you hit the limit

Cross the concurrency ceiling and Crawlbase responds with HTTP 429 Too Many Requests. The request is rejected — it's not queued — so you should retry with backoff.

// HTTP/1.1 429 Too Many Requests
// Retry-After: 1
// X-Crawlbase-Concurrency: 20

{ "error": "Concurrency limit reached", "limit": 20 }

The Retry-After header tells you the minimum seconds before retrying. Always honor it.

Handling rate limits gracefully

The right pattern is exponential backoff with jitter. Most HTTP clients have this built in; here's the bare minimum if you need to roll your own.

import time, random
from crawlbase import CrawlingAPI

api = CrawlingAPI({'token': 'YOUR_TOKEN'})

def crawl_with_retry(url, attempts=5):
    for i in range(attempts):
        res = api.get(url)
        if res['status_code'] != 429:
            return res
        wait = (2 ** i) + random.random()
        time.sleep(wait)
    raise RuntimeError('Rate limit exhausted')
const { CrawlingAPI } = require('crawlbase');
const api = new CrawlingAPI({ token: 'YOUR_TOKEN' });

async function crawlWithRetry(url, attempts = 5) {
  for (let i = 0; i < attempts; i++) {
    const res = await api.get(url);
    if (res.statusCode !== 429) return res;
    const wait = (2 ** i) * 1000 + Math.random() * 1000;
    await new Promise(r => setTimeout(r, wait));
  }
  throw new Error('Rate limit exhausted');
}

For workloads that consistently push the ceiling, use a worker pool instead of fire-and-forget. Cap the pool at your concurrency limit and you'll never see a 429.

Scaling beyond the defaults

Three options when 20 concurrent isn't enough:

Higher concurrency

We routinely set tokens to 100, 500, or 1,000+ concurrent for customers with the volume to justify it. Email support with your target throughput and use case.

Multiple tokens

free

Each Crawlbase account can have multiple sub-accounts, each with its own token and concurrency budget. Use this to isolate workloads.

Enterprise Crawler

async

Push URLs to a managed queue and let Crawlbase handle concurrency, retries, and delivery to your webhook. No client-side scheduling required.

Best practices

Cap worker pools at your token limit. Don't oversubscribe and rely on 429s — the rejected request still takes a round-trip.
Use async=true for slow targets. Long-running JS-rendered crawls block a concurrency slot for the entire request. Async mode releases the slot immediately and delivers the result via webhook.
Always honor Retry-After. Ignoring it just creates more 429s and burns network round-trips.
Add jitter to retries. If 50 workers all retry at exactly 1s after a transient spike, you'll just spike again.
Alert on sustained 429 rate. An occasional 429 is fine. A sustained 5%+ rate means you need higher concurrency or smarter scheduling.