AI coding assistants scaffold APIs, generate frontend components, and write tests in seconds. But a model can only reason over what it already knows or what you paste into context, and that breaks down the moment you integrate an unfamiliar API, work against documentation that changed yesterday, or build on a fast-moving platform.
Ask Codex to integrate a new service and it will confidently generate the client, wire up authentication, and set the environment variables. Everything looks right until you check it against the official docs: a required parameter is missing, an endpoint was renamed, auth now needs an extra header. None of it is obvious until runtime, because the generated code still looks plausible.
The usual fix is to break your flow: open the docs in another tab, dig through GitHub for the latest README, paste the relevant sections into chat, and ask for a regeneration until everything lines up. The friction isn't writing code. It's shuttling between research and implementation.
The Crawlbase Codex Plugin removes that split. Backed by the Crawlbase Web MCP Server, it gives Codex live access to the web through natural language: crawl HTML, extract clean Markdown, capture screenshots, and optionally persist pages in Crawlbase Cloud Storage, all without leaving your editor. A companion repository ships the prompts, sample files, and project structure used here. Clone it before you start:
git clone https://github.com/ScraperHub/turn-codex-into-a-full-stack-web-scraper.git cd turn-codex-into-a-full-stack-web-scraper
By the end you will have installed the plugin, verified it with a smoke test, pulled live documentation straight from GitHub, and had Codex build a minimal full-stack scraper from that documentation. More usefully, you will have a workflow that applies to almost any AI-assisted task where accurate, current information matters.
Why AI-assisted development needs live web data
Models are strong at reasoning over the context they are given. Their weak spot is not programming, it is information that changed after training. Documentation evolves constantly: SDKs add parameters, APIs deprecate endpoints, authentication requirements shift, and release notes carry implementation details that never make it into tutorials.
Without live web access, the loop is manual: ask Codex to integrate an API, get mostly-correct code, compare it against the current docs, paste the differences back into chat, and regenerate. With the plugin, Codex closes that loop itself.
Instead of gathering documentation by hand, Codex retrieves the current version before it writes a line of code.
What the Crawlbase Codex Plugin is
The OpenAI Codex plugins ecosystem lets you extend Codex with reusable skills, app integrations, and MCP servers. The Crawlbase Codex Plugin packages the Crawlbase Web MCP Server into a plugin that exposes production crawling through natural language. Once installed, you can ask Codex to:
- Crawl a webpage and return HTML.
- Extract clean Markdown from an article.
- Capture full-page screenshots.
- Crawl JavaScript-rendered sites.
- Store crawled pages in Crawlbase Cloud Storage, and retrieve them later without crawling again.
Under the hood the plugin launches npx -y @crawlbase/mcp, passes your Crawlbase credentials through environment variables, and talks to Crawlbase's crawling infrastructure. The hero diagram above traces the whole path: your prompt goes to Codex, Codex calls the plugin, the plugin drives the Web MCP Server, and Crawlbase handles rendering, retries, proxy rotation, anti-bot, and storage behind the scenes. From where you sit, it all happens through prompts.
You could ask Codex to write scraping code from scratch, but it would be guessing at a moving target: the site's markup, its anti-bot posture, and the current API shape. The plugin routes every request through the Crawling API, so Codex inherits JavaScript rendering, residential IP rotation, and anti-bot handling, and gets clean content back on the first call.
What you'll build
The goal is not just to scrape one site, but to build a repeatable engineering workflow that pairs live documentation with AI-assisted development. Over the next five steps you will install the plugin, validate it with a smoke test, fetch the latest Web MCP documentation from GitHub, use that documentation to generate a minimal full-stack scraper, and learn where screenshots and Cloud Storage fit into larger workflows.
Step 1: Install the plugin
Clone the plugin into Codex's plugins directory:
git clone https://github.com/crawlbase/crawlbase-codex-plugin ~/.codex/plugins/crawlbase-mcp
Export your Crawlbase API tokens (if you don't have an account yet, sign up for Crawlbase first). The companion repo includes a .env.example listing the variables it expects:
export CRAWLBASE_TOKEN=your_token_here export CRAWLBASE_JS_TOKEN=your_js_token_here
Restart Codex. The plugin then appears as Crawlbase Web Scraper, exposing tools like crawl, crawl_markdown, crawl_screenshot, and the Cloud Storage operations. The bundled .mcp.json launches npx -y @crawlbase/mcp and reads your credentials from the environment, so no secrets live inside the plugin itself, which keeps it safe for shared, version-controlled environments.
Step 2: Verify with a smoke test
Before generating application code, confirm the whole pipeline works. Paste prompts/00-smoke-test.txt into Codex, or simply ask:
Get the markdown content of https://example.com
The request does almost nothing on purpose. If everything is wired up, Codex calls crawl_markdown and returns clean Markdown (title, headings, body) instead of raw HTML. That confirms the full execution path, from your prompt through Codex, the plugin, and the Web MCP Server, down to the Crawling API. That path is the backbone of the rest of the tutorial: you describe the outcome, and the server feeds live web content into Codex's context. If it fails, check that CRAWLBASE_TOKEN is exported and restart Codex after installing the plugin.
Step 3: Fetch the latest docs before writing code
Rather than have Codex integrate Crawlbase from memory, give it the current documentation from GitHub first. Paste prompts/01-fetch-api-readme.txt into Codex; it retrieves the Web MCP README from the raw GitHub URL, covering environment variables, the MCP tools, store=true, Cloud Storage retrieval, and token-specific storage behavior.
Training data goes stale. Pulling the README straight from GitHub means Codex works from the same documentation that is on the main branch today. If the README is large, crawl it with store=true and pull it back later with storage_get using as=markdown (see the Cloud Storage docs).
Step 4: Generate the full-stack scraper
With the current documentation in context, ask Codex to generate the app using prompts/02-build-full-stack-scraper.txt. The prompt asks for three layers:
| Layer | Responsibility |
|---|---|
| Backend |
POST /scrape, calls the Crawlbase Crawling API |
| Frontend | URL input, submit button, results panel |
| Project | README, .env.example, .gitignore, run instructions |
Codex now has your requirements, your project context, and the latest documentation, so it generates against the current API reference rather than historical training data. Test the app against books.toscrape.com or another public practice site. If the returned Markdown is empty because the site depends on JavaScript, retry with CRAWLBASE_JS_TOKEN, following the documentation you retrieved in Step 3.
Step 5: Iterate with live web data
Open prompts/03-iterate-with-live-data.txt. It bundles workflows for comparing Markdown output, capturing screenshots, storing documentation with store=true, and retrying JavaScript-rendered pages. At this point research, implementation, and debugging all live inside a single Codex conversation instead of scattered across tabs.
Persist crawls with Cloud Storage
Many scraping workflows don't end at a single request. You may need to revisit documentation, compare how a page changed, or reprocess the same crawl without hitting the target site again. Set store: true on any crawl and, instead of returning the page inline, Crawlbase keeps the underlying HTML in Cloud Storage and returns metadata such as rid, stored_at, storage_url, and token_type. You manage the stored content with:
-
storage_getandstorage_bulk_get -
storage_listandstorage_count -
storage_deleteandstorage_bulk_delete
One detail is worth knowing: whether you call crawl, crawl_markdown, or crawl_screenshot, the stored artifact is always the underlying HTML. When you retrieve a stored page as Markdown, Crawlbase generates it from that HTML at retrieval time. Screenshots are not kept in Cloud Storage; they come back separately through a temporary screenshot URL.
Understanding storage partitions
Cloud Storage partitions crawled pages by the API token used for the crawl. Pages fetched with CRAWLBASE_TOKEN live in a different silo from pages fetched with CRAWLBASE_JS_TOKEN, because standard crawls and JavaScript-rendered crawls run on different infrastructure.
token_type of normal or js; pass that same value (via use_js_token) to the storage tools so you query the partition the page actually landed in.Every successful crawl response includes a token_type field (normal or js) telling you where the document was stored. Reuse that value for later storage operations: if a page was crawled with the JavaScript token, pass use_js_token: true to storage_get, storage_list, or storage_bulk_get. A "Not found" response usually doesn't mean the document is gone; it means you are querying the wrong partition. Checking the original crawl's token_type is normally enough to find the right silo.
Engineering best practices
The workflow above is great for prototypes; a few practices make it hold up as projects grow.
- Fetch current docs before generating integration code. APIs and SDKs move fast, and giving the model today's documentation sharply cuts down on deprecated endpoints, wrong parameters, and stale auth flows.
-
Keep credentials in environment variables. Commit a
.env.exampleso others know what is required, but never commit real tokens. -
Store large crawls instead of inlining them. For big documentation pages or datasets,
store: truekeeps the conversation focused and lets you retrieve the content later. - Treat generated code like production code. Validate status codes, set sensible timeouts, and surface meaningful errors. AI-assisted or not, robust error handling is what makes an app reliable.
As projects grow past prototypes, concurrency, retries, monitoring, and infrastructure reliability start to matter too; our guide on scaling web scraping projects covers those production concerns.
Conclusion
AI coding assistants are at their best when they can reason over current information instead of historical training data. Pairing the Crawlbase Codex Plugin with the Crawlbase Web MCP Server gives Codex direct access to live web content: it can retrieve documentation, extract clean Markdown, capture screenshots, and build applications from the latest available information, all inside one conversation.
Give Codex (and any MCP client) live web access in a single tool call. Every crawl renders JavaScript behind a rotating residential IP and returns clean Markdown, with optional Cloud Storage for reuse. No proxy pool, no headless fleet, no scraping code to maintain. Get your API tokens and build on the free tier.
Frequently asked questions
What is the Crawlbase Codex Plugin?
It's an extension for OpenAI Codex that integrates the Crawlbase Web MCP Server into your development workflow. Once installed, Codex can crawl webpages, extract clean Markdown, capture screenshots, and manage Crawlbase Cloud Storage through natural-language prompts. Instead of writing scraping infrastructure yourself, you ask Codex to retrieve live web content and use it as context for code generation.
Why use the Web MCP Server instead of asking Codex to generate a scraper?
Codex is good at generating scraping code, but it has no access to live sites or documentation unless you supply it. The plugin gives Codex real-time web access through Crawlbase's infrastructure, so it can pull the latest API docs, crawl sites, render JavaScript pages, and generate code from current information rather than potentially outdated training data.
Can I use the plugin for JavaScript-heavy websites?
Yes. It supports both static and JavaScript-rendered pages. With CRAWLBASE_JS_TOKEN configured, Codex uses Crawlbase's JavaScript rendering to crawl dynamic sites, extract content after scripts run, and capture rendered screenshots.
Why does storage_get return "Not found" after storing a page?
Pages crawled with CRAWLBASE_TOKEN are stored separately from pages crawled with CRAWLBASE_JS_TOKEN. If a page was crawled with the JavaScript token, pass use_js_token: true to the storage tools. Most "Not found" responses just mean you are querying the wrong storage partition, not that the page is missing.
Crawl any site at scale, without fighting infrastructure.
Crawlbase handles proxies, fingerprints, and CAPTCHAs so your team ships data pipelines instead of maintaining crawl plumbing. 1,000 requests free, no card required.