# Crawlbase developer documentation The web, structured for builders. Crawl, scrape, and parse any website at scale with a single API. Production-ready endpoints, native SDKs, and an MCP server that plugs straight into Claude, Cursor, and your agent stack. [Get started](/docs/quick-start)[API Reference](/docs/api-reference)[MCP for AI](/docs/ai) Up to 5,000 free requests195 countries51M req/month per tokenNo credit card ~/crawlbase $ curl'https://api.crawlbase.com/?' \'token=YOUR\_TOKEN'\'&url=https://github.com/crawlbase' → 200 OK // 4.2s · cb\_status: 200 · 14.8 KB# JS-rendered, geo-routed, anti-bot bypassed \\…\ $ curl'https://api.crawlbase.com/?' \'token=YOUR\_TOKEN&format=json'\'&url=https://github.com/crawlbase' → {"original\_status": 200,"cb\_status": 200,"url": "https://github.com/crawlbase","body": "\…"} $ curl'https://api.crawlbase.com/?' \'token=YOUR\_TOKEN&format=md'\'&url=https://github.com/crawlbase' → # CrawlbaseWeb crawling & scraping API - Python, Node.js, Ruby, PHP, Go SDKs. # Or via the MCP server (same result, agent-native)\> tool\_use:crawl\_markdown(url="https://github.com/crawlbase") HTMLJSONMarkdown APIs ## Pick the surface that fits your stack [Browse all APIs](/docs/api-reference) [ Crawling API General purpose crawling with full headless-browser rendering, residential proxies, and built-in anti-bot bypass. The Swiss army knife. JSONHTMLMarkdown ](/docs/crawling-api)[ Enterprise Crawler Push millions of URLs at high concurrency, get results streamed back to your webhook. We handle the queueing, retries, and storage. WebhooksAsyncQueues ](/docs/crawler)[ Smart AI Proxy Residential & datacenter proxies with rotation. A single endpoint that picks the right exit node, retries failures, works with any HTTP client. HTTPSOCKS5Sticky IP ](/docs/smart-proxy)[ Cloud Storage Store, manage and serve scraped data. Persist crawled HTML and parsed JSON - fetch later by URL or RID, no infrastructure to operate. S3 CompatibleCDN ](/docs/cloud-storage) Use cases ## What can I build? [Browse scrapers](/docs/scrapers) [ Price & availability monitoring Poll Amazon, Walmart, Best Buy or any retailer's product pages on a schedule. Snapshot price, stock, and rating fields into a database - alert when they move. E-commerce scrapersCrawler queues ](/docs/scrapers/ecommerce)[ SEO & rank tracking Daily SERP snapshots for your target keywords. Track domain position in organic, watch People-Also-Ask coverage, build a SERP-feature presence dashboard. Google SERPBing SERP ](/docs/scrapers/search-engines)[ AI agents & training data Real-time web access for Claude, Cursor, and any MCP-compatible agent. Or batch-crawl a corpus and pipe Markdown into a retrieval index for grounded LLM answers. MCP serverMarkdown export ](/docs/ai)[ Lead enrichment & prospecting Walk a list of company domains, pull every visible email address with the email-extractor scraper, enrich with LinkedIn company / profile data. Lead lists ready for CRM upsert. email-extractorLinkedIn scrapers ](/docs/scrapers/email-extractor)[ Competitor & brand monitoring Track competitor product launches, social engagement, and review sentiment over time. Diff scraped JSON week-over-week to flag pricing, copy, or feature changes. Social scrapersGeneric extractor ](/docs/scrapers/social-media) Quickstart ## Your first crawl in 60 seconds [ 1 Grab your token Sign up free, no credit card. You'll get a Normal token (TCP) and a JavaScript token. ](/login)[ 2 Make your first request Encode a URL, hit the endpoint with curl or your SDK. The response includes the crawled page plus metadata. ](/docs/crawling-api#quickstart)[ 3 Scale up Switch to async mode, push to a Crawler queue, or wire up the MCP server to your AI agent. ](/docs/crawler) GEThttps://api.crawlbase.com/ ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN&url=https%3A%2F%2Fgithub.com%2Fcrawlbase' ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) response = api.get('https://github.com/crawlbase') if response['status_code'] == 200: print(response['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); api.get('https://github.com/crawlbase') .then(res => console.log(res.statusCode, res.body)) .catch(err => console.error(err)); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') response = api.get('https://github.com/crawlbase') puts response.status_code puts response.body ``` ``` 'YOUR_TOKEN']); $response = $api->get('https://github.com/crawlbase'); echo $response->statusCode; echo $response->body; ``` ``` package main import ( "fmt" "github.com/crawlbase/crawlbase-go" ) func main() { api := crawlbase.NewCrawlingAPI("YOUR_TOKEN") res, _ := api.Get("https://github.com/crawlbase") fmt.Println(res.StatusCode, res.Body) } ``` AI & MCP ## Native plumbing for AI agents [Explore AI & MCP](/docs/ai) [Crawlbase MCP Server](/docs/ai-mcp) Expose every Crawlbase tool to Claude, Cursor, ChatGPT, and any MCP-compatible client. Read docs [Claude Desktop](/docs/ai-claude) One-click install in Claude. Crawlbase becomes a native tool your conversations can call. Setup [LangChain & LlamaIndex](/docs/integrations-langchain) Crawlbase document loaders, retrievers, and tools - drop-in for your agent graph. Integrate [Prompt patterns](/docs/ai-prompts) Battle-tested prompts for extraction, summarization, monitoring, and lead enrichment. Browse SDKs & Integrations ## Drop into any stack [ ![Python logo](/assets/images/python-logo-round-6275a9b1c7.png) Python pip install crawlbase ](/docs/sdk-python)[ ![Node.js logo](/assets/images/node-js-logo-round-dcd8bf0db6.png) Node.js npm i crawlbase ](/docs/sdk-node)[ ![Ruby logo](/assets/images/ruby-logo-round-42f0935ab8.png) Ruby gem install crawlbase ](/docs/sdk-ruby)[ ![PHP logo](/assets/images/php-logo-round-8d5ebe870d.png) PHP composer require crawlbase ](/docs/sdk-php)[ Go Golang go get crawlbase-go ](/docs/sdk-go)[ ![Zapier logo](/assets/images/zapier-logo-round-33c40eb7c3.png) Zapier no-code automation ](/docs/integrations-zapier)[ ![n8n logo](/assets/images/n8n-circle-2715fc8444.png) n8n self-hosted workflows ](/docs/integrations-n8n)[ Make visual scenarios ](/docs/integrations-make) [Status & Errors](/docs/status-codes) Every code, every meaning. Bookmark this page. [Rate Limits](/docs/rate-limits) 20 req/sec per token, with paths to higher limits. [Changelog](/docs/changelog) What shipped this week, with migration notes. [Talk to engineering](/docs/support) Got a hard problem? We're a Slack/email away.