Product / Cloud Storage

Cloud Storage.
Every crawl kept, none re-fetched.

Add store=true to any crawl and Crawlbase keeps the rendered page, JSON and screenshots in the cloud.
Retrieve by RID, run full-text search, and skip the database, the S3 bucket and the backups.

Start freeRead the docs
10,000 documents freeOne flag: store=trueFull-text search
Crawl and storePull anytimecrawl any URLstore=trueCloud StorageStoreIndexSearchRetrieve by RIDFull-text searchPages and shotsGET /storagesearch your crawlsHTML and screenshotscrawled · stored to cloud · 200
Live storage writes1.24M req/minStreaming
200zillow.com/homes/for_sale/IN189ms
200indeed.com/jobs?q=developerJP43ms
200walmart.com/ip/55048794ES109ms
200linkedin.com/jobs/searchGB104ms
200tripadvisor.com/Restaurants-g60763AU94ms
200ebay.com/itm/204512389011US147ms
404glassdoor.com/Reviews/index.htmDE98ms
200amazon.com/dp/B08N5WRWNWJP124ms
200github.com/crawlbaseGB54ms
200reddit.com/r/programmingCA73ms
200tripadvisor.com/Restaurants-g60763AU191ms
404ebay.com/itm/204512389011BR141ms
200walmart.com/ip/55048794BR201ms
200tripadvisor.com/Restaurants-g60763AU110ms
200glassdoor.com/Reviews/index.htmAU75ms
200walmart.com/ip/55048794GB79ms
200booking.com/searchresults.html?ss=ParisJP201ms
200producthunt.com/posts/notionAU145ms
200github.com/crawlbaseNL139ms
200linkedin.com/jobs/searchAU130ms
200walmart.com/ip/55048794DE177ms
200linkedin.com/jobs/searchDE83ms
200tripadvisor.com/Restaurants-g60763SG63ms
200walmart.com/ip/55048794AU189ms
200indeed.com/jobs?q=developerJP160ms
200producthunt.com/posts/notionCA95ms
200zillow.com/homes/for_sale/IN189ms
200indeed.com/jobs?q=developerJP43ms
200walmart.com/ip/55048794ES109ms
200linkedin.com/jobs/searchGB104ms
200tripadvisor.com/Restaurants-g60763AU94ms
200ebay.com/itm/204512389011US147ms
404glassdoor.com/Reviews/index.htmDE98ms
200amazon.com/dp/B08N5WRWNWJP124ms
200github.com/crawlbaseGB54ms
200reddit.com/r/programmingCA73ms
200tripadvisor.com/Restaurants-g60763AU191ms
404ebay.com/itm/204512389011BR141ms
200walmart.com/ip/55048794BR201ms
200tripadvisor.com/Restaurants-g60763AU110ms
200glassdoor.com/Reviews/index.htmAU75ms
200walmart.com/ip/55048794GB79ms
200booking.com/searchresults.html?ss=ParisJP201ms
200producthunt.com/posts/notionAU145ms
200github.com/crawlbaseNL139ms
200linkedin.com/jobs/searchAU130ms
200walmart.com/ip/55048794DE177ms
200linkedin.com/jobs/searchDE83ms
200tripadvisor.com/Restaurants-g60763SG63ms
200walmart.com/ip/55048794AU189ms
200indeed.com/jobs?q=developerJP160ms
200producthunt.com/posts/notionCA95ms
01 Live demo

Store with a flag. Pull by RID.

Cloud Storage, typed live. Store a crawl with one parameter, then retrieve it later by its RID. Hover to pause and read.

ready
keys 1-2 switch · click to pauserun your own URL
Store your first crawl in minutes. 10,000 documents free, no credit card.Start free
02 Capabilities

A store built for crawls.

Keep, find and pull your crawled data without standing up storage of your own.

store

Store with one flag

Add store=true to any Crawling API call, or point a Crawler at cloud storage, and the rendered page is kept automatically.

retrieve

Retrieve by RID

Every stored crawl gets a RID. Send a GET request with it and the page comes straight back, no re-crawl required.

search

Full-text search

Search across everything you have crawled to find the exact pages you need, instead of scanning your own database.

capture

Pages and screenshots

Keep rendered HTML, structured JSON and full-page screenshots together, all retrievable by the same RID.

scale

Scaling handled

Crawlbase manages scaling, backups and cleanup of your space, so you can retire the S3 bucket and the maintenance.

pipe

Crawler webhook target

Use storage as the delivery target for the async Crawler, so large crawls land ready to pull.

03 How it works

One flag to store, one call to pull.

Keep your crawl as it is. Add a parameter to store, and a GET to retrieve.

01

Add store=true

Crawl any URL with store=true, or set cloud storage as your Crawler webhook target.

02

We store and index

The rendered page, JSON and screenshot are stored and indexed for full-text search, with scaling handled.

03

Save the RID

The response returns a storage RID that uniquely identifies the page you just kept.

04

Retrieve or search

Send a GET to the storage endpoint with the RID, search across your crawls, or pull from the dashboard.

04 Use cases

What teams keep in Cloud Storage.

USE / 01Pipelines

A buffer for your stack

Land crawled pages in storage and pull them into your warehouse, index or model on your own schedule.

USE / 02History

Snapshots over time

Keep every crawl so you can compare a page across dates without crawling it again.

USE / 03Search

Find across your crawls

Run full-text search over everything stored to locate the exact pages and fields you need.

USE / 04Cost

Retire your own storage

Drop the S3 bucket and database. Storage scales, backs up and cleans itself for you.

USE / 05AI

Corpora for training and RAG

Build and re-pull large, clean page sets for training and retrieval straight from storage.

USE / 06Async

Crawler delivery

Pair storage with the Crawler so high-volume async crawls arrive ready to retrieve.

05 Pricing

Free to start, cheap to scale.

Cloud Storage is free for developers and inexpensive for companies. You pay for crawl volume, not for a database to run.

FreeStart building AI and web data workflows
$0/ mo
Start free
  • 10k requests
  • 14 days retention
  • Get API
  • Delete API
  • Bulk API
DeveloperReliable storage for AI and automation workflows
$29/ mo
Choose Developer
  • 100k requests
  • 30 days retention
  • Get API
  • Delete API
  • Bulk API
Most popular
BusinessBuilt for scalable data operations
$249/ mo
Choose Business
  • 1M requests
  • 30 days retention
  • Get API
  • Delete API
  • Bulk API
EnterpriseCustom infrastructure for large-scale data systems
Contact us
Contact us
  • Custom volume
  • Custom retention
  • Get API
  • Delete API
  • Bulk API
06 Why Crawlbase

Storage you do not have to run.

Cloud Storage runs on the same network that serves 70,000+ developers. No S3 to provision, no database to back up, nothing to clean up when it fills.

99%
Average request success rate
70K+
Customers on the network
10K
Documents free for 14 days
99.99%
Network uptime

Add store=true once, and every crawl is kept, indexed and ready to pull.

07 FAQ

Cloud Storage questions.

A scalable cloud store for your crawled and scraped data. Add store=true to a crawl and the rendered HTML, JSON and screenshots are kept for you, ready to retrieve by RID or search, with no database of your own to run.
Add store=true to your Crawling API call, or use Cloud Storage as the webhook target for a Crawler. The response returns a storage RID that identifies the stored page.
Send a GET request to the storage endpoint with the RID, run a full-text search across your stored pages, or browse and download from the storage dashboard.
Yes. Stored crawls can include the rendered HTML, structured JSON and full-page screenshots, all retrievable by the same RID.
No. Crawlbase handles scaling, backups and cleanup of your storage space, so you can stop maintaining an S3 bucket or a database of your own.
It is free for developers, storing up to 10,000 documents for 14 days with no credit card. Beyond that it scales inexpensively with your crawl volume, and the same token works across the Crawling API, the Crawler and every scraper.

Keep every page you crawl.
Skip the database.

Free to begin, store up to 10,000 documents for 14 days. One token across Cloud Storage and every product.

Start freeRead the docs