Product / Cloud Storage

Cloud Storage.
Every crawl kept, none re-fetched.

Add store=true to any crawl and Crawlbase keeps the rendered page, JSON and screenshots in the cloud.
Retrieve by RID, run full-text search, and skip the database, the S3 bucket and the backups.

Start free Read the docs

0.5 credits per page per monthOne flag: store=trueFull-text search

Live storage writes1.24M req/minStreaming

200tripadvisor.com/Restaurants-g60763SG108ms

200google.com/search?q=web+scrapingSG73ms

200target.com/p/-/A-79404211NL108ms

200tripadvisor.com/Restaurants-g60763IN76ms

200zillow.com/homes/for_sale/CA77ms

200target.com/p/-/A-79404211IN100ms

200github.com/crawlbaseUS118ms

200google.com/search?q=web+scrapingFR117ms

200indeed.com/jobs?q=developerUS196ms

200booking.com/searchresults.html?ss=ParisAU79ms

200ebay.com/itm/204512389011NL91ms

200ebay.com/itm/204512389011NL124ms

200linkedin.com/jobs/searchES136ms

200yelp.com/biz/blue-bottle-coffeeJP194ms

200yelp.com/biz/blue-bottle-coffeeBR215ms

200yelp.com/biz/blue-bottle-coffeeGB88ms

200zillow.com/homes/for_sale/FR205ms

200booking.com/searchresults.html?ss=ParisBR217ms

200amazon.com/dp/B08N5WRWNWNL75ms

200walmart.com/ip/55048794SG153ms

200google.com/search?q=web+scrapingGB62ms

200yelp.com/biz/blue-bottle-coffeeIN126ms

200zillow.com/homes/for_sale/FR57ms

200linkedin.com/jobs/searchJP147ms

200producthunt.com/posts/notionSG101ms

200google.com/search?q=web+scrapingNL59ms

200tripadvisor.com/Restaurants-g60763SG108ms

200google.com/search?q=web+scrapingSG73ms

200target.com/p/-/A-79404211NL108ms

200tripadvisor.com/Restaurants-g60763IN76ms

200zillow.com/homes/for_sale/CA77ms

200target.com/p/-/A-79404211IN100ms

200github.com/crawlbaseUS118ms

200google.com/search?q=web+scrapingFR117ms

200indeed.com/jobs?q=developerUS196ms

200booking.com/searchresults.html?ss=ParisAU79ms

200ebay.com/itm/204512389011NL91ms

200ebay.com/itm/204512389011NL124ms

200linkedin.com/jobs/searchES136ms

200yelp.com/biz/blue-bottle-coffeeJP194ms

200yelp.com/biz/blue-bottle-coffeeBR215ms

200yelp.com/biz/blue-bottle-coffeeGB88ms

200zillow.com/homes/for_sale/FR205ms

200booking.com/searchresults.html?ss=ParisBR217ms

200amazon.com/dp/B08N5WRWNWNL75ms

200walmart.com/ip/55048794SG153ms

200google.com/search?q=web+scrapingGB62ms

200yelp.com/biz/blue-bottle-coffeeIN126ms

200zillow.com/homes/for_sale/FR57ms

200linkedin.com/jobs/searchJP147ms

200producthunt.com/posts/notionSG101ms

200google.com/search?q=web+scrapingNL59ms

01 Live demo

Store with a flag. Pull by RID.

Cloud Storage, typed live. Store a crawl with one parameter, then retrieve it later by its RID. Hover to pause and read.

ready

keys 1-2 switch · click to pauserun your own URL

Store your first crawl in minutes. Start free, no credit card.Start free

02 Capabilities

A store built for crawls.

Keep, find and pull your crawled data without standing up storage of your own.

store

Store with one flag

Add store=true to any Crawling API call, or point a Crawler at cloud storage, and the rendered page is kept automatically.

retrieve

Retrieve by RID

Every stored crawl gets a RID. Send a GET request with it and the page comes straight back, no re-crawl required.

Full-text search

Search across everything you have crawled to find the exact pages you need, instead of scanning your own database.

capture

Pages and screenshots

Keep rendered HTML, structured JSON and full-page screenshots together, all retrievable by the same RID.

scale

Scaling handled

Crawlbase manages scaling, backups and cleanup of your space, so you can retire the S3 bucket and the maintenance.

pipe

Crawler webhook target

Use storage as the delivery target for the async Crawler, so large crawls land ready to pull.

03 How it works

One flag to store, one call to pull.

Keep your crawl as it is. Add a parameter to store, and a GET to retrieve.

Add store=true

Crawl any URL with store=true, or set cloud storage as your Crawler webhook target.

We store and index

The rendered page, JSON and screenshot are stored and indexed for full-text search, with scaling handled.

Save the RID

The response returns a storage RID that uniquely identifies the page you just kept.

Retrieve or search

Send a GET to the storage endpoint with the RID, search across your crawls, or pull from the console.

04 Use cases

What teams keep in Cloud Storage.

USE / 01Pipelines

A buffer for your stack

Land crawled pages in storage and pull them into your warehouse, index or model on your own schedule.

USE / 02History

Snapshots over time

Keep every crawl so you can compare a page across dates without crawling it again.

USE / 03Search

Find across your crawls

Run full-text search over everything stored to locate the exact pages and fields you need.

USE / 04Cost

Retire your own storage

Drop the S3 bucket and database. Storage scales, backs up and cleans itself for you.

USE / 05AI

Corpora for training and RAG

Build and re-pull large, clean page sets for training and retrieval straight from storage.

USE / 06Async

Crawler delivery

Pair storage with the Crawler so high-volume async crawls arrive ready to retrieve.

05 Pricing

One rate card, shared with every product.

Storage bills from the same balance as your crawls: half a credit keeps a page stored for a month, with no database to run.

Pay as you go$3.00 down to $0.02 per 1,000 successful requests

Only successful requests are billed, and rates fall as volume grows. Optional subscriptions from $99 / mo. Start free with up to 5,000 requests, no credit card.

See pricing Start free

06 Why Crawlbase

Storage you do not have to run.

Cloud Storage runs on the same network that serves 70,000+ developers. No S3 to provision, no database to back up, nothing to clean up when it fills.

99%

Average request success rate

70K+

Customers on the network

0.5

Credits per stored page per month

99.99%

Network uptime

Add store=true once, and every crawl is kept, indexed and ready to pull.

07 FAQ

Cloud Storage questions.

A scalable cloud store for your crawled and scraped data. Add store=true to a crawl and the rendered HTML, JSON and screenshots are kept for you, ready to retrieve by RID or search, with no database of your own to run.

Add store=true to your Crawling API call, or use Cloud Storage as the webhook target for a Crawler. The response returns a storage RID that identifies the stored page.

Send a GET request to the storage endpoint with the RID, run a full-text search across your stored pages, or browse and download from the Storage page in the console.

Yes. Stored crawls can include the rendered HTML, structured JSON and full-page screenshots, all retrievable by the same RID.

No. Crawlbase handles scaling, backups and cleanup of your storage space, so you can stop maintaining an S3 bucket or a database of your own.

Storage draws on the same credit balance as your crawls: half a credit keeps a page stored for a month, and your free signup credits cover it to start. See the pricing page for how credits work.

Keep every page you crawl.
Skip the database.

Free to begin, no credit card. One balance across Cloud Storage and every product.