Changelog · Crawlbase Documentation

May 2026

Relaunched /docs end-to-end — restructured navigation, dark mode, and instant in-page transitions across every section.
Command palette search (⌘K / Ctrl K) jumps straight to pages, sections, and API parameters from anywhere.
New Ask AI button opens a Crawlbase-trained assistant in-page so you can ask anything about the docs without leaving them.
Interactive API Playground runs real Crawling API calls right in the browser, with response headers and a rendered page preview alongside the body.
Append .md to any docs URL (e.g. /docs/crawling-api.md) to get a clean Markdown copy you can hand to an LLM.
Published /llms.txt as an index for AI-assistant discovery.
Refreshed German, French, Russian, and Simplified Chinese translations across every page for sharper, more idiomatic copy.

First official Go SDK for the Crawlbase API. Single CrawlingAPI client, dependency-free, idiomatic Go.
Source: github.com/crawlbase/crawlbase-go. Reference docs published on pkg.go.dev.

langchain-crawlbase is now on PyPI — a document loader, tool, and retriever backed by the Crawling API.
Source: github.com/crawlbase/langchain-crawlbase.

New format=md parameter on the Crawling API returns clean Markdown instead of HTML.
Pair it with md_readability=true to strip nav, ads, and chrome before conversion — same idea as Reader Mode.

Crawling API device parameter now accepts tablet alongside desktop and mobile.

The Crawlbase MCP server gains six storage tools so agents can list, read, and clean up Cloud Storage items between crawls — not just trigger new ones.
Published as @crawlbase/[email protected].

The Crawlbase Codex plugin brings Crawlbase MCP into OpenAI Codex.
Manual install today via git clone into ~/.codex/plugins/; Codex Marketplace listing in review.
Source: github.com/crawlbase/crawlbase-codex-plugin

Enterprise Crawler push now accepts queue_timeout, so you can cap how long a request sits in queue before it's dropped instead of waiting forever.

The asynchronous push/pull product is now branded Enterprise Crawler across the dashboard and docs to distinguish it from the synchronous Crawling API.
Endpoints, parameters, and tokens are unchanged.

The MCP server now supports header-based authentication and an optional HTTP transport mode in addition to stdio — useful for shared/remote MCP setups.
Published as @crawlbase/[email protected].

The Crawlbase rename is now reflected in every translated edition of the docs (previously English-only).

The retrieval-and-retention product is now Cloud Storage across docs, dashboard, and SDK method names. Existing /storage endpoints continue to work.

Crawling API gains custom_success_code so you can mark non-2xx responses as successful when scraping endpoints that legitimately return e.g. 404 or 451.

Smart Proxy is now branded Smart AI Proxy, reflecting the AI-driven routing and retry logic that's been added underneath. No client-side changes.

First public release of the Crawlbase MCP server as @crawlbase/mcp on npm — three crawl tools (crawl, crawl_markdown, crawl_screenshot) usable from Claude Desktop & Claude Code, Cursor, and VS Code / Windsurf.

Smart Proxy now forwards custom request headers and cookies through to the target site — useful for authenticated crawls and session-pinned scraping.

Crawling API docs now spell out exactly how scroll_interval counts toward billing on long-scroll pages, so you can predict the cost of an infinite-scroll crawl before issuing it.