The Best MCP Servers for Web ScrapingNine Options Compared for 2026

An MCP server lets an AI assistant such as Claude, Cursor or an agent built on the OpenAI Agents SDK call outside tools in the middle of a conversation. For web scraping, that means the model can fetch a live page, read it as Markdown, take a screenshot or run a search without you writing glue code. Most scraping vendors now ship one, and they differ a lot in what they expose, where they run, and, most of all, how well they get through sites that block bots.

This guide compares the MCP servers for web scraping worth testing in 2026, what each is actually good at, and how to connect the Crawlbase Web MCP Server to Claude Desktop and Claude Code. Every server detail below was rechecked against the vendor's own documentation or repository on 29 September 2026. MCP servers change quickly, so confirm tool names and limits before you build on them.

What an MCP server for web scraping does

The Model Context Protocol is an open standard for connecting AI assistants to external tools. A scraping MCP server registers a few tools, such as "fetch this URL as Markdown", and the client decides when to call them. The server does the network work and returns the result to the model as text or an image.

Two consequences follow. The quality of the answer is capped by the quality of the fetch, so a server that fetches from your own IP will fail exactly where a plain script fails. And every tool the server registers is described to the model on every turn, so a server with dozens of tools spends more of the context window before any work starts than one with a handful.

What to compare

1. Success on protected sites. Does the server run on a proxy network with anti-bot handling, or fetch from your machine? Test it on the sites you actually need.
2. JavaScript rendering. Can it load single-page apps and late content, and does it render every call or only when asked?
3. Output format. Clean Markdown is the cheapest format for a model to read. Raw HTML is useful for parsing but expensive in tokens.
4. Hosted or local. A hosted endpoint needs no install. A local server launched with npx or uvx keeps the token in a process you control.
5. Tool count. A broad server covers more jobs but adds context to every turn. Check whether tool groups can be switched off.
6. Billing. Free server plus metered API calls, credits with multipliers, or open source with no service behind it, and whether failed requests are billed.
7. Client support. Copy-paste configs for the clients your team uses: Claude Desktop, Claude Code, Cursor, VS Code or Windsurf.

The servers

1. Crawlbase Web MCP Server

Best for: assistants that need to read live, JavaScript-heavy or protected pages on the same infrastructure a production scraper would use.

The Crawlbase Web MCP Server sends every call through the Crawling API, so proxies and anti-bot handling apply throughout, and JavaScript rendering is used when a call needs it: screenshots, page waits, device emulation and scrolling. It registers three crawl tools:

• crawl fetches a URL and returns its HTML.
• crawl_markdown returns clean Markdown extracted from the page, the cheapest format for a model to read.
• crawl_screenshot returns a JPEG of the page, full page by default, that the model can see directly.

Alongside them sit six storage tools for pages you save with store: true, so v1.3.0 registers nine tools in all. The server is open source on GitHub and runs with npx @crawlbase/mcp@latest on Node 18 or later; Crawlbase also hosts it at https://mcp.crawlbase.com/mcp, where one key is enough. It works with Claude Desktop, Claude Code, Cursor, Windsurf and n8n.

Billing: the server itself is free. You pay for the Crawling API requests it makes, and failed requests are not billed. A rendered request costs twice the site's tier, a screenshot costs 2 credits, and a page saved to Cloud Storage costs half a credit a month. New accounts start with up to 5,000 free requests, no card.

2. Bright Data MCP

Best for: agents that need search, scraping, structured data from large sites and browser automation from one server.

Bright Data's open-source MCP server (MIT) is the broadest here: search across Google, Bing and Yandex, scrape_as_markdown and batch scraping, a large family of web_data_* tools for structured data from sites such as Amazon and LinkedIn, and 13 browser automation tools. Use the hosted endpoint at mcp.brightdata.com or run it locally with npx @brightdata/mcp. The free allowance is 5,000 requests a month, no card.

Watch out for: the full tool set is large. The README documents GROUPS and TOOLS settings to load only what you need, and it is worth using them.

3. Firecrawl MCP

Best for: crawling and mapping whole sites from inside an assistant, with search and extraction.

The Firecrawl MCP server (MIT) exposes firecrawl_scrape, firecrawl_map, firecrawl_search, firecrawl_crawl, parse and agent tools, and newer research and monitoring tools, 26 in the full profile. It is hosted at mcp.firecrawl.dev, runs locally with npx -y firecrawl-mcp, or points at a self-hosted Firecrawl. Scrape, search and parse work without a key at a rate limit; crawl, map and agent need one.

Watch out for: the self-hosted Firecrawl build leaves out Fire-engine, its anti-bot layer, so a self-hosted MCP setup may not reach protected sites.

4. Apify MCP

Best for: agents that should pick a ready-made scraper for a specific site from a large marketplace.

The Apify MCP server (MIT) connects an assistant to the thousands of Actors in Apify Store. Its default tools search for an Actor, fetch its details and call it, so the model can find a scraper for a site and run it. It is hosted at mcp.apify.com with OAuth or a bearer token, runs locally with npx @apify/actors-mcp-server, and works with Claude Code, Claude.ai, Cursor and VS Code.

Watch out for: quality and cost depend on the Actor the model picks. For production jobs, pin the Actors you trust instead of letting the model search the whole store.

5. Oxylabs MCP

Best for: teams already on Oxylabs that want its scraper APIs and AI Studio tools in an assistant.

Oxylabs recommends its hosted endpoint at mcp.oxylabs.io, with tools for the Web Scraper API (a universal scraper plus Google and Amazon scrapers) and AI Studio tools for scraping, crawling, browsing, search and site mapping. The GitHub repository is archived, but Oxylabs' docs still document a local install with uvx oxylabs-mcp.

6. ScrapingBee MCP

Best for: teams on ScrapingBee that want HTML, text, screenshots and a few site-specific tools in an assistant.

ScrapingBee runs a remote MCP server at mcp.scrapingbee.com, with tools for page HTML and text, screenshots, file downloads, Google search, and Amazon, Walmart and YouTube. Claude Desktop and Cursor connect through mcp-remote, and calls use ScrapingBee credits at the same rate as the API.

Watch out for: the only documented setup passes the API key in the endpoint URL, so treat the MCP config file as a secret.

7. ScraperAPI MCP

Best for: structured data from search, retail and real estate sites, plus async crawl jobs.

The ScraperAPI MCP server (MIT) registers 28 tools: a general scrape tool with rendering, country and output options, structured data tools for Google, Amazon, Walmart, eBay and Redfin, crawler job tools and AI parser tools. It runs remotely or locally as a Python package (Python 3.11 or later) or in Docker.

Watch out for: with 28 tools the context cost is real. If you only need page content, a smaller server is cheaper per turn.

8. Jina MCP

Best for: reading public pages and papers, search and reranking for research-style agents.

Jina's remote MCP server (Apache-2.0) wraps Reader and Search: read_url and capture_screenshot_url for pages, search_web and search_arxiv for search, and sort_by_relevance and deduplicate_strings for ranking. The reading tools work without a key at a rate limit; search and ranking need one. It fits documentation and articles well. Reader is the fetch behind it, and Jina's own FAQ says Reader does not bypass anti-bot systems.

9. Playwright MCP

Best for: clicking through flows, filling forms and testing your own web app from an assistant.

Microsoft's Playwright MCP (Apache-2.0) drives a real local browser through accessibility snapshots and runs with npx @playwright/mcp@latest. It is free. It accepts a proxy server but documents no unblocking, so on protected sites you supply the proxies. Its own README suggests coding agents may get better token efficiency from Playwright CLI with skills.

Also worth knowing: Scrapy MCP

Zyte announced Scrapy MCP on 23 September 2026. It does not fetch pages: it connects an agent such as Claude Code to a running Scrapy crawl, so the agent can inspect jobs and debug the spider live. If your team writes Scrapy spiders, it pairs well with any fetcher above.

The servers compared

Connect the Crawlbase Web MCP Server to Claude

You need a Crawlbase account and your tokens from the dashboard: the Normal token for plain requests and the JavaScript token for rendered pages and screenshots. The examples use placeholders; never paste real tokens into shared files or screenshots.

Claude Desktop

Open Settings, then Developer, then Edit Config, add this to claude_desktop_config.json, and restart Claude Desktop:

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "YOUR_TOKEN",
        "CRAWLBASE_JS_TOKEN": "YOUR_JS_TOKEN"
      }
    }
  }
}

Claude Code

Claude Code reads the same mcpServers shape from a project's .mcp.json, or from ~/.claude.json for your user. The block above runs the server locally. To use the hosted server instead, with nothing to install, point it at the endpoint and send one token as a bearer header:

{
  "mcpServers": {
    "crawlbase": {
      "type": "streamable-http",
      "url": "https://mcp.crawlbase.com/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_TOKEN"
      }
    }
  }
}

Once connected, ask for something that needs live data, such as "Read the pricing page at this URL and summarize the plans as a table." Claude calls crawl_markdown, reads the result and answers. The launch post and the MCP documentation cover the rest.

Two security notes. With the local server, the client sees tool definitions and results but never your tokens; with the hosted setup, the token sits in the client's config file, so treat that file as a secret. Either way the model can request any URL, so use a token you can rotate and watch your request log.

How to choose

• Your assistant needs live pages, including protected and JavaScript-heavy ones: a server on a scraping network with proxies and rendering, such as the Crawlbase Web MCP Server.
• You want search, scraping and site-specific data from one server: Bright Data covers the most ground, if you trim its tool groups.
• You need to crawl or map whole sites from the assistant: Firecrawl's crawl and map tools.
• You want a ready-made scraper for one site: Apify, with the Actors pinned.
• You are automating a browser flow or testing your own app: Playwright MCP, with your own proxies if the target blocks bots.
• You are debugging Scrapy spiders: add Scrapy MCP next to your fetcher.

Whatever you shortlist, run the same 20 to 50 real URLs through each server and compare three things: how many come back with real content, how clean the output is, and how much context each turn costs.

Frequently Asked Questions (FAQs)

What is an MCP server for web scraping?

A small server that speaks the Model Context Protocol and exposes scraping actions, such as fetching a page as Markdown or taking a screenshot, as tools an AI assistant can call. The assistant decides when to call them and reads the result.

Which MCP server is best for Claude?

Every server in this guide works with Claude. For reading live pages, including protected ones, use a server backed by a scraping API with proxies and rendering, such as the Crawlbase Web MCP Server, so the model gets real content instead of block pages. For browser flows, use Playwright MCP.

Is the Crawlbase Web MCP Server free?

The server is free and open source. You pay for the Crawling API requests it makes, and failed requests are not billed; a screenshot costs 2 credits, and saving a page to Cloud Storage costs half a credit a month. New accounts start with up to 5,000 free requests, no credit card.

Do I need to install anything?

Not always. Crawlbase, Bright Data, Firecrawl, Apify, Oxylabs, ScrapingBee, ScraperAPI and Jina all offer hosted endpoints. Local servers usually run on demand through npx or uvx, so there is no global install either.

Can an MCP server scrape sites that block bots?

Only if it runs on infrastructure built for it. A server that fetches from your own machine, such as a local browser automation server, gets blocked the same way a script would. Servers backed by a scraping API route requests through proxies and handle anti-bot challenges server-side.

VP

Viktor Petrov

Data Science Architect · Crawlbase

Data Science Architect at Crawlbase. Works on the models that keep large-scale crawling reliable against modern bot defenses.