Docs
Log in

AI & MCP

Plug Crawlbase into your AI agent stack. The MCP server gives any MCP-compatible client (Claude Desktop, Cursor, VS Code, OpenAI's Codex agent) live, structured access to the web — no model can browse without it, with it the model can crawl, scrape, and screenshot any URL on demand.

What MCP is

MCP (Model Context Protocol) is an open spec from Anthropic for connecting AI assistants to external tools and data. An MCP server exposes a set of named tools (like crawl_url, crawl_markdown, crawl_screenshot) that any MCP-aware client can discover and call. The Crawlbase MCP server gives your agent the same crawl + scrape capabilities the Crawling API provides — exposed as agent-callable tools instead of REST endpoints.

Start here

  • ai-mcp — MCP server reference. Install steps, tools exposed (with input/output schemas), client config, environment variables, security notes. The most thorough page in this section — read it first.

Use it with your client

Per-client setup guides for the IDEs and agents that already speak MCP:

  • ai-claude — Claude Desktop and Claude Code. Drop one block of JSON into your settings; Claude can crawl URLs you reference in chat.
  • ai-cursor — Cursor. The MCP server appears as a tool the agent can call mid-conversation.
  • ai-vscode — VS Code (with Continue, Claude Code, or any MCP-aware extension).
  • ai-codex new — OpenAI Codex agent. Adds Crawlbase as a connected tool.
  • ai-opencode new — OpenCode terminal agent. Native MCP client (local + remote, with OAuth) — drops Crawlbase tools straight into the in-shell coding loop.

Prompt patterns

  • ai-prompts — recipes for asking the model to use the Crawlbase tools effectively. RAG retrieval, comparison flows, multi-step research, screenshot-grounded reasoning — proven prompt shapes for each.

Why route AI through Crawlbase

Most LLM clients can't browse the live web at all; the few that can (search-augmented chat, browser-using agents) hit the same walls human scrapers do — JS-heavy SPAs, anti-bot challenges, geo-restricted content, rate limits. The MCP server hands the agent a tool that has already solved those problems: every Crawlbase feature (residential proxies, JS rendering, bot bypass, geo-routing, the scraper library) is reachable from the agent without exposing the model to any of the implementation detail.

If you're shipping an agent product, this is the cheapest way to give it real-world web access without writing your own browser-automation layer.