Changelog

September 2026

2026-09-15 · Console: Viewer role

• Workspaces gain a Viewer role for read-only access. A viewer can open Usage, Requests, Storage and Crawlers, but never sees an API key or a crawler destination and cannot change anything. Pick the role for a member in Settings > Team.

2026-09-06 · Crawlbase Cookbook

• The Crawlbase Cookbook is live: a page for each of 356 sites our customers crawl, opening with figures measured on real Crawlbase traffic, such as success rate, whether a browser is needed and credits per page. Find a site by search, filters or 26 category pages, follow step-by-step recipes such as Sephora and idealo, and read how we count.

2026-09-06 · Playground: one key for JavaScript rendering

• Code snippets in the console Playground and the docs Playground now use your normal key with javascript=true for rendered requests, instead of the JavaScript key. The API switches to your JavaScript key before the crawl, so one key covers both, and the request is billed as a JavaScript request. See JavaScript parameters.

2026-09-06 · Console: card expiry and failed payment notices

• Billing shows the card on file and its expiry date. Within 60 days of expiry the note turns amber, and red once the card has expired, so you can update it before the next charge. If a card payment fails and is being retried, the console shows a Payment failed banner with the amount and a link to update your payment method.

2026-09-05 · Console: overage in dollars

• On subscription plans, overage is now shown in dollars as well as credits on Billing, the console home page and Usage, priced the same way as your invoice. Billing also shows your next renewal: the plan plus the expected overage.

2026-09-05 · Console: scraper catalogue

• Scrapers lists every scraper as a card with a short description, a link to its docs and a Try in Playground button. Filter by name or category.

2026-09-01 · Console: full invoice history

• The invoice list on Billing now covers your whole history, page by page, and exports to CSV. Each invoice downloads as the document that was sent to you.

August 2026

2026-08-25 · Console: leave an organization, change owners

• In Settings > Team, members can now leave an organization themselves, and owners can make another member an owner, or an owner a member, by clicking their role. The last owner cannot be removed. If you accept a workspace invite while you have your own subscription, the console first explains that your subscription keeps billing on your own organization.

2026-08-25 · LangChain docs: source and package links

• The LangChain integration page now links to the source on GitHub and the package on PyPI, so you can check both before installing langchain-crawlbase.

2026-08-22 · Console: crawler search and live board

• Crawlers has a search box that matches crawler names as you type, plus All, Running and Paused filters; a crawler paused for low balance counts as paused. Each crawler's live tab refreshes every 5 seconds and shows queued and in-flight jobs and the oldest wait.

2026-08-22 · Console: usage for the billing period, with an overage calculator

• Usage now opens on This period: your plan's billing period, or the calendar month to date without a plan, so the totals match what you are billed. Subscription accounts see the plan credits used first, and the projection slider works as an overage calculator, showing which volumes your plan covers and what overage would cost beyond it.

2026-08-21 · Console: archive workspaces, usage by workspace

• Owners can archive a workspace in Settings > Team. Its tokens stop working at once; its usage and billing history stay visible. On accounts with two or more workspaces, selecting a workspace scopes Usage and the home chart to that workspace's traffic.

2026-08-20 · Console: readable storage previews

• Storage previews are now formatted: JSON bodies, including scraper output and format=json results, are indented, and HTML pages are laid out one tag per line instead of one long line. The stored data is unchanged.

2026-08-19 · Console: cancel or keep your plan

• The billing owner can cancel a self-serve plan from Billing. The plan stays active until the end of the period and is not renewed. Until then it shows its end date, and Keep plan undoes the cancellation.

2026-08-14 · The new console

• The dashboard is rebuilt as the console. Requests is a searchable log of your API calls, with status filters, a detail view per request and CSV export. Usage breaks credits down by product, tier and domain. Also included: Storage, Crawlers with a delivery log and retries, the Playground, support tickets, Settings and Billing with invoices and card payments. Workspaces let you split keys and usage by team or project.

2026-08-11 · Proxy mode retired

• The Crawling API can no longer be called as an HTTP proxy. The proxy-mode listener has been retired and its reference section removed from these docs. Smart AI Proxy remains the proxy-interface product and covers the same capabilities, with its own subscription and its own concurrency model. REST calls to the Crawling API are unaffected.

2026-08-08 · Crawling API: async on every domain

• async=true now works against any target. It was previously accepted only for linkedin.com URLs, with other domains enabled per token on request. The call returns an rid straight away and the finished page is delivered to your callback webhook, or stored and fetched later from Cloud Storage by the same rid. See Async & storage.

2026-08-07 · New scrapers: LeetCode

• Four new scrapers for LeetCode, in the Developer & Tech category - LeetCode Problem Set (a problem set listing as an array with id, difficulty, acceptance rate, and premium flag), LeetCode Problem (a single problem with its statement, topic tags, hints, and starter code snippets), LeetCode Solutions (the community solutions tab as an array with author, tags, and engagement counts), and LeetCode Solution (a single solution post with its write-up and extracted code blocks).

July 2026

2026-07-29 · New scrapers: Kaggle

• Four new scrapers for Kaggle, in the Developer & Tech category - Kaggle Dataset Search (a dataset search or listing page as a ranked array with owner, size, usability rating, and downloads), Kaggle Dataset (a single dataset with its description, keywords, license, and file list), Kaggle Notebook Search (a notebook search or listing page as a ranked array with author, co-authors, and competition context), and Kaggle Notebook (a single notebook with its language, runtime, version history, and attached inputs).

2026-07-24 · New scrapers: OLX

• Two new scrapers for OLX classifieds, in the E-Commerce category, across the shared frontend (olx.pl, olx.ua, olx.pt, olx.ro, olx.bg, olx.kz, olx.uz) - OLX SERP (a search or category results page as a ranked array of listings with pricing and location, plus pagination) and OLX Item (a single ad with its full attributes, price, seller, and images).

2026-07-22 · New scrapers: Exercism

• Four new scrapers for Exercism, in the Developer & Tech category - Exercism Exercises (a track exercises page as a structured array with difficulty and unlock state), Exercism Exercise (a single exercise with its full instructions as text and HTML), Exercism Solutions (an exercise community-solutions page as a paginated array of authors, languages, and stars), and Exercism Solution (a single published solution with its iteration history and full source code).

2026-07-20 · New scrapers: Stack Exchange

• Two new scrapers for the Stack Exchange network (Stack Overflow, Super User, Ask Ubuntu, Server Fault, MathOverflow, and every *.stackexchange.com site), in the Reviews & Q&A category - Stack Exchange Questions (a questions, tagged, or search-results page as a structured array with pagination) and Stack Exchange Thread (a single question with its full answer and comment threads).

2026-07-16 · New scrapers: Booking.com

• Two new scrapers for Booking.com, in the Travel, Events & Real Estate category - Booking SERP (a search-results listing as a structured array of properties with pricing and review scores, plus pagination) and Booking Hotel (a single hotel page with pricing, review scores, and facilities).

2026-07-16 · New scrapers: Product Hunt

• Two new scrapers for Product Hunt, in the Reviews & Q&A category - Product Hunt Leaderboard (daily and weekly leaderboards as a ranked array of products) and Product Hunt Product (a single product page with upvotes, makers, topics, and reviews).

2026-07-15 · New scrapers: Reddit

• Three new scrapers for Reddit, in the Social Media category - Reddit Subreddit (a subreddit listing as a ranked array of posts with pagination), Reddit Search (a Reddit search-results page as a structured array with pagination), and Reddit Post (a single post with its full comment tree).

2026-07-14 · New scrapers: GitHub

• Three new scrapers for GitHub, grouped under the new Developer category - GitHub Repository (a single repository page: stars, forks, watchers, languages, topics, license, open issues and PRs, default branch, and latest release), GitHub SERP (a repository search-results page as a structured array with pagination), and GitHub Profile (a user or organization profile: bio, followers, following, public repos, pinned repos, and organizations).

June 2026

2026-06-30 · New scrapers: Google Trends

• Two new scrapers for Google Trends - Google Trends (the "Trending now" page, returning the top trending searches with search-volume metrics and trend breakdowns) and Google Trends Explore (returning interest over time, interest by sub-region, related topics, and related queries for a given topic).

2026-06-29 · Crawler API: delete a crawler

• New POST /crawler/<TOKEN>/<NAME>/delete endpoint deletes a crawler entirely - it clears the queue and unregisters the crawler, so it no longer shows up in your dashboard or stats. Unlike Purge, which only empties the queue, delete removes the crawler itself. This action is permanent.

2026-06-06 · New scrapers: Galaxus

• Three new scrapers for Galaxus - Product, SERP, and Product Reviews - returning structured product, search, and review data from galaxus.ch.

2026-06-04 · Crawling API: PDF output

• New pdf=true parameter on the Crawling API returns the fully rendered page as a PDF (Content-Type: application/pdf) instead of HTML.
• Combine it with the rendering parameters (country, device, page_wait); PDF requests are billed the same as a JavaScript-rendered request.

2026-06-02 · Crawler management API

• Create, retrieve, and update Crawlers programmatically over REST, instead of only through the dashboard.
• New endpoints: POST /crawler/<TOKEN> to create, GET /crawler/<TOKEN>/<NAME> for details, and PUT /crawler/<TOKEN>/<NAME> to update.

May 2026

2026-05-21 · Passwordless onboarding

• Sign in with a one-time code emailed to you. No password to set, remember, or reset; existing accounts continue to work as before.
• Redesigned onboarding for new accounts. We pre-select a workflow (scraping, MCP, or integrations) from your stated use case, show a live preview of your first Crawling API request, and route you into the matching dashboard section when you finish.

2026-05-08 · New docs site

• Relaunched /docs end-to-end - restructured navigation, dark mode, and instant in-page transitions across every section.
• Command palette search (⌘K / Ctrl K) jumps straight to pages, sections, and API parameters from anywhere.
• New Ask AI button opens a Crawlbase-trained assistant in-page so you can ask anything about the docs without leaving them.
• Interactive API Playground runs real Crawling API calls right in the browser, with response headers and a rendered page preview alongside the body.
• Append .md to any docs URL (e.g. /docs/crawling-api.md) to get a clean Markdown copy you can hand to an LLM.
• Published /llms.txt as an index for AI-assistant discovery.
• Refreshed German, French, Russian, and Simplified Chinese translations across every page for sharper, more idiomatic copy.

2026-05-03 · Go SDK v0.1.0

• First official Go SDK for the Crawlbase API. Single CrawlingAPI client, dependency-free, idiomatic Go.
• Source: github.com/crawlbase/crawlbase-go. Reference docs published on pkg.go.dev.

2026-05-02 · LangChain integration v0.1.0

• langchain-crawlbase is now on PyPI - a document loader, tool, and retriever backed by the Crawling API.
• Source: github.com/crawlbase/langchain-crawlbase.

April 2026

2026-04-24 · Crawling API: Markdown output

• New format=md parameter on the Crawling API returns clean Markdown instead of HTML.
• Pair it with md_readability=true to strip nav, ads, and chrome before conversion - same idea as Reader Mode.

2026-04-23 · Tablet device option

• Crawling API device parameter now accepts tablet alongside desktop and mobile.

2026-04-23 · MCP Server v1.3.0 - storage tools

• The Crawlbase MCP server gains six storage tools so agents can list, read, and clean up Cloud Storage items between crawls - not just trigger new ones.
• Published as @crawlbase/[email protected].

2026-04-23 · Codex plugin

• The Crawlbase Codex plugin brings Crawlbase MCP into OpenAI Codex.
• Manual install today via git clone into ~/.codex/plugins/; Codex Marketplace listing in review.
• Source: github.com/crawlbase/crawlbase-codex-plugin

March 2026

2026-03-25 · Enterprise Crawler: queue_timeout

• Enterprise Crawler push now accepts queue_timeout, so you can cap how long a request sits in queue before it's dropped instead of waiting forever.

February 2026

2026-02-10 · Crawler renamed to Enterprise Crawler

• The asynchronous push/pull product is now branded Enterprise Crawler across the dashboard and docs to distinguish it from the synchronous Crawling API.
• Endpoints, parameters, and tokens are unchanged.

November 2025

2025-11-25 · MCP Server v1.2.0 - auth & HTTP mode

• The MCP server now supports header-based authentication and an optional HTTP transport mode in addition to stdio - useful for shared/remote MCP setups.
• Published as @crawlbase/[email protected].

2025-11-04 · Crawlbase brand across all locales

• The Crawlbase rename is now reflected in every translated edition of the docs (previously English-only).

October 2025

2025-10-20 · Storage API renamed to Cloud Storage

• The retrieval-and-retention product is now Cloud Storage across docs, dashboard, and SDK method names. Existing /storage endpoints continue to work.

September 2025

2025-09-26 · custom_success_code parameter

• Crawling API gains custom_success_code so you can mark non-2xx responses as successful when scraping endpoints that legitimately return e.g. 404 or 451.

2025-09-16 · Smart Proxy → Smart AI Proxy

• Smart Proxy is now branded Smart AI Proxy, reflecting the AI-driven routing and retry logic that's been added underneath. No client-side changes.

July 2025

2025-07-10 · Crawlbase MCP Server v1.0

• First public release of the Crawlbase MCP server as @crawlbase/mcp on npm - three crawl tools (crawl, crawl_markdown, crawl_screenshot) usable from Claude Desktop & Claude Code, Cursor, and VS Code / Windsurf.

June 2025

2025-06-25 · Smart Proxy: header & cookie forwarding

• Smart Proxy now forwards custom request headers and cookies through to the target site - useful for authenticated crawls and session-pinned scraping.

2025-06-18 · scroll_interval billing clarified

• Crawling API docs now spell out exactly how scroll_interval counts toward billing on long-scroll pages, so you can predict the cost of an infinite-scroll crawl before issuing it.