Crawlbase developer documentation
The web,
structured for builders.
Crawl, scrape, and parse any website at scale with a single API. Production-ready endpoints, native SDKs, and an MCP server that plugs straight into Claude, Cursor, and your agent stack.
Pick the surface that fits your stack
General purpose crawling with full headless-browser rendering, residential proxies, and built-in anti-bot bypass. The Swiss army knife.
Push millions of URLs at high concurrency, get results streamed back to your webhook. We handle the queueing, retries, and storage.
Residential & datacenter proxies with rotation. A single endpoint that picks the right exit node, retries failures, works with any HTTP client.
Store, manage and serve scraped data. Persist crawled HTML and parsed JSON - fetch later by URL or RID, no infrastructure to operate.
What can I build?
Poll Amazon, Walmart, Best Buy or any retailer's product pages on a schedule. Snapshot price, stock, and rating fields into a database - alert when they move.
Daily SERP snapshots for your target keywords. Track domain position in organic, watch People-Also-Ask coverage, build a SERP-feature presence dashboard.
Real-time web access for Claude, Cursor, and any MCP-compatible agent. Or batch-crawl a corpus and pipe Markdown into a retrieval index for grounded LLM answers.
Walk a list of company domains, pull every visible email address with the email-extractor scraper, enrich with LinkedIn company / profile data. Lead lists ready for CRM upsert.
Track competitor product launches, social engagement, and review sentiment over time. Diff scraped JSON week-over-week to flag pricing, copy, or feature changes.
Your first crawl in 60 seconds
curl 'https://api.crawlbase.com/?token=YOUR_TOKEN&url=https%3A%2F%2Fgithub.com%2Fcrawlbase'from crawlbase import CrawlingAPI
api = CrawlingAPI({'token': 'YOUR_TOKEN'})
response = api.get('https://github.com/crawlbase')
if response['status_code'] == 200:
print(response['body'])const { CrawlingAPI } = require('crawlbase');
const api = new CrawlingAPI({ token: 'YOUR_TOKEN' });
api.get('https://github.com/crawlbase')
.then(res => console.log(res.statusCode, res.body))
.catch(err => console.error(err));require 'crawlbase'
api = Crawlbase::API.new(token: 'YOUR_TOKEN')
response = api.get('https://github.com/crawlbase')
puts response.status_code
puts response.body<?php
use Crawlbase\CrawlingAPI;
$api = new CrawlingAPI(['token' => 'YOUR_TOKEN']);
$response = $api->get('https://github.com/crawlbase');
echo $response->statusCode;
echo $response->body;package main
import (
"fmt"
"github.com/crawlbase/crawlbase-go"
)
func main() {
api := crawlbase.NewCrawlingAPI("YOUR_TOKEN")
res, _ := api.Get("https://github.com/crawlbase")
fmt.Println(res.StatusCode, res.Body)
}
