Web scraping changes character once a workload approaches a billion requests a month. At small scale, the engineering problem is extracting data from a website. At enterprise scale, the problem moves to operations: keeping refresh pipelines on schedule, holding success rates steady, absorbing anti-bot pressure, and making sure the collection layer never becomes the bottleneck for everything downstream.
This case study follows how a leading U.S.-based vacation rental intelligence platform integrated the Crawlbase Enterprise Crawler into an existing data platform to support roughly one billion monthly requests across Airbnb, Vrbo, Booking.com, and regional travel marketplaces.
The company's analytics platform was already built to ingest, normalize, and analyze large volumes of marketplace data. Rather than redesigning that system, the engineering team replaced only the collection layer. Crawlbase took over the job of acquiring web data reliably at scale, which left the customer free to keep working on what actually differentiated the product: turning raw marketplace data into business intelligence.
5.52 billion successful requests, averaging 919 million per month and roughly 30.6 million per day, at a 99.96% average success rate. Monthly volume grew 51% from November 2025 to a March 2026 peak of 1.04 billion, and monthly success never dropped below 99.78%.
The workload
The customer's analytics platform serves more than 2,300 professional lodging organizations across 220+ countries and territories, combining public marketplace data with direct reservation data to produce real-time market intelligence for the short-term rental industry.
Unlike an analytics platform built on a single source, their system continuously merges two independent streams. The first comes from 65+ property management system (PMS) integrations, covering reservation and operational data for roughly 700,000 managed properties. The second comes from large-scale collection of public marketplace data across Airbnb, Vrbo, Booking.com, and regional vacation rental platforms.
| Metric | Value |
|---|---|
| Geographic coverage | 220+ countries and territories |
| PMS integrations | 65+ |
| Managed properties | 700,000+ |
| Airbnb listings monitored | 6.15 million |
| Vrbo listings monitored | 1.85 million |
| Daily crawl requests at full coverage | ~33 million |
| Monthly crawl requests | ~1 billion |
Every crawl feeds a calculation: occupancy, availability, nightly pricing, booking lead times, ADR, RevPAR, and competitive benchmarks across thousands of local markets. Refreshing listings, calendars, pricing, and availability across millions of properties is what pushes the daily figure to roughly 33 million requests, and the monthly figure toward a billion.
At that scale the engineering challenge changes shape. Collecting one listing page is straightforward. Collecting millions of them continuously, finishing every refresh cycle inside a strict SLA window, and delivering the data fast enough that downstream analytics stay current is a different problem entirely.
Reliability starts to matter more than peak throughput. Every delayed crawl, timeout, or incomplete job eventually surfaces as stale market intelligence, and that shows up in the occupancy trends, pricing benchmarks, and forecasting models that thousands of professional lodging teams depend on.
When the collection layer becomes the bottleneck
Workloads this size expose problems that rarely appear in smaller scraping systems.
As daily volume climbed toward 33 million requests, the collection infrastructure started to strain. Crawl latency rose during sustained workloads, so large refresh jobs took progressively longer to finish. Request timeouts became more frequent, particularly on dynamic listing pages and availability calendars from Airbnb and Vrbo. Individual failures were manageable; across millions of requests they accumulated into incomplete batches and late refresh cycles.
Running self-managed proxy infrastructure at that scale was its own problem. Routing performance varied between regions, intermittent network failures introduced unpredictable delays, and keeping proxy pools healthy demanded constant attention. Meanwhile Airbnb and Vrbo kept strengthening their anti-bot defenses, cutting effective crawl yield through blocking, fingerprinting, and other detection.
The consequences reached well past the scraping layer. Every delayed crawl meant downstream analytics were working from older marketplace data. Availability calendars drifted, pricing snapshots fell behind real market conditions, and competitive benchmarks lost freshness whenever a refresh window slipped past its target.
The largest cost was engineering time. Instead of building analytics capabilities, the team spent more and more of its week maintaining crawler infrastructure, tuning proxy pools, recovering failed jobs, and chasing collection issues. None of that was a bug to be fixed. It was the natural consequence of running a collection layer at a scale where predictable reliability matters more than raw throughput.
Replacing only the collection layer
The analytics platform was never the problem. Ingestion, normalization, enrichment, and reporting already handled large volumes of marketplace and reservation data comfortably. What could not keep pace was the layer responsible for retrieving that information from the public web.
So the team isolated the bottleneck and replaced only crawl execution. The Enterprise Crawler took over everything required to retrieve web data reliably at scale:
- Request routing.
- Rotating datacenter and residential proxy infrastructure.
- JavaScript rendering for dynamic pages.
- Automatic retries and failure recovery.
- Network resilience.
- Anti-bot handling across the supported marketplaces.
Everything after collection stayed exactly as it was. Once Crawlbase retrieved a page, the customer's existing pipeline carried it through validation, normalization, enrichment, and analytics unchanged.
Separating collection from downstream processing simplified the architecture. Crawlbase absorbed the operational complexity of request execution, proxy rotation, retries, rendering, and anti-bot resilience, while the customer's infrastructure stayed focused on turning raw web data into market intelligence.
The split also let each layer scale on its own schedule. As crawl volume grew, improvements to collection no longer required changes to parsing, storage, or analytics. Those are the same principles we cover in our guide to large-scale web scraping architecture, applied to a production workload.
Scaling toward a billion requests a month
The clearest test of a crawl infrastructure is how it behaves as the workload grows. Plenty of systems handle moderate traffic well and then need architectural surgery once volume climbs.
Over the six-month period, monthly successful volume grew from roughly 687 million requests in November 2025 to a peak of 1.04 billion in March 2026, an increase of about 51%. April closed at 990.7 million, still 44% above where the window started. The trend line across the period works out to roughly +52.6 million requests per month.
| Month | Successful requests | Success rate |
|---|---|---|
| Nov 2025 | 686.7M | 100.00% |
| Dec 2025 | 889.8M | 100.00% |
| Jan 2026 | 1,015.8M | 100.00% |
| Feb 2026 | 894.8M | 99.98% |
| Mar 2026 | 1,037.5M | 99.78% |
| Apr 2026 | 990.7M | 100.00% |
Across the full window that comes to 5.52 billion successful requests, an average of 919 million per month, and roughly 30.6 million per day. Note the distinction between the two daily figures: about 33 million requests a day is what a complete refresh cycle costs at full coverage, while 30.6 million is the measured daily average across six months of real traffic.
Those numbers are more than throughput. Every successful crawl feeds parsing, enrichment, and analytics systems that produce occupancy trends, pricing intelligence, availability calendars, ADR, RevPAR, and competitive benchmarks across millions of monitored properties. The trend shows the collection layer absorbing additional workload without becoming the limiting factor, and without a redesign of the downstream pipeline.
Reliability under sustained load
Scaling volume is only half the problem. A platform that can process billions of requests is only useful if the results stay dependable while the workload moves around.
Over the same six months, success stayed remarkably flat despite significant swings in monthly traffic:
- 99.96% average request success rate.
- Monthly success between 99.78% and 100.00%, a range of 0.22 points.
- Close to a billion successful requests a month through changing traffic patterns.
Success rate translates directly into analytics quality. Every completed request is another piece of marketplace data that can be validated, parsed, enriched, and folded into occupancy models, pricing benchmarks, availability calendars, and forecasting dashboards. Failures leave gaps, and gaps reduce both freshness and completeness.
The most useful observation is how little reliability moved while throughput grew by more than 50%. The collection layer did not trade stability for volume. For a continuously refreshed analytics platform, that consistency is often worth more than a higher peak, because finishing every collection cycle on schedule is what keeps downstream systems processing fresh data instead of compensating for infrastructure noise.
Operational impact
Because every downstream system depended on marketplace data arriving on time, improvements at the collection stage propagated through the whole pipeline.
More reliable data pipelines
The immediate change was consistency. Crawl jobs completed predictably across Airbnb, Vrbo, Booking.com, and regional marketplaces, so refresh cycles finished inside their processing windows even as volume kept climbing.
Downstream ingestion, parsing, and normalization spent less time compensating for delayed or incomplete batches and more time processing fresh data. More consistent refreshes meant more current occupancy trends, nightly pricing, ADR, RevPAR, availability calendars, booking lead times, and competitive benchmarks. The outcome was not simply faster scraping; it was more dependable business intelligence.
Lower operational overhead
Offloading collection also cut the engineering effort needed to run large-scale scraping infrastructure. Time that had gone into troubleshooting crawler failures, managing proxies, and recovering incomplete jobs went into expanding analytics coverage and supporting new markets instead.
As monitored inventory, customer adoption, and crawl volume grew, the collection layer kept supporting roughly a billion monthly requests without turning back into a recurring engineering problem.
Takeaways
Four principles from this deployment apply to any organization building large-scale web data systems.
- Separate collection from downstream processing. Collection, transformation, storage, and analytics have different scaling characteristics. Decoupling them lets each evolve independently and reduces operational complexity.
- Optimize for predictable completion, not peak throughput. For a continuously refreshed platform, finishing every cycle reliably beats occasional performance spikes.
- Treat anti-bot resilience as shared infrastructure. At enterprise scale, routing, proxy management, retries, and rendering are platform capabilities, not features of individual scrapers.
- Measure operational health, not just volume. Throughput alone says little. Success rates, completion consistency, retry behavior, and refresh latency show whether a pipeline is actually delivering.
As workloads grow into the hundreds of millions or billions of requests, success depends less on individual scrapers and more on treating data collection as production infrastructure. A collection layer that scales lets engineering teams spend their time on the products and insights that create value instead of on the crawlers underneath them.
The managed collection layer behind this deployment: request routing, rotating datacenter and residential proxies, JavaScript rendering, automatic retries, and anti-bot handling, delivered asynchronously to your webhook or Cloud Storage. Billions of requests a month without a crawler fleet to operate. Talk to us about enterprise volume, or start on the free tier.
Frequently asked questions
What does a billion crawl requests a month actually involve?
For this platform it works out to roughly 33 million requests a day at full coverage, refreshing listings, calendars, pricing, and availability across 6.15 million Airbnb listings, 1.85 million Vrbo listings, and additional inventory on Booking.com and regional marketplaces. Across six measured months it came to 5.52 billion successful requests, averaging 919 million a month.
Why replace only the collection layer instead of the whole pipeline?
Because the rest of the pipeline was not the bottleneck. Ingestion, normalization, enrichment, and reporting already handled the volume. Isolating crawl execution meant the team could fix the failing layer without touching parsing, storage, or analytics, and each layer could then scale on its own schedule.
Does reliability drop as crawl volume increases?
It did not here. Monthly volume grew more than 50% across the period while the success rate stayed between 99.78% and 100.00%, a 0.22 point band, averaging 99.96%. For continuously refreshed analytics, that consistency matters more than peak throughput, because a missed refresh window shows up as stale market data.
How is anti-bot handling managed at this scale?
As shared infrastructure rather than per-scraper logic. Request routing, proxy rotation across datacenter and residential pools, JavaScript rendering for dynamic pages, and automatic retries all sit inside the Enterprise Crawler, so individual collection jobs do not carry their own bypass code and do not need updating each time a marketplace changes its defenses.
What should teams measure to know whether their collection layer is healthy?
Success rate, completion consistency, retry behavior, and refresh latency, not request volume on its own. Throughput says nothing about whether refresh cycles are finishing on time. The useful question is whether every cycle completes inside its window at a stable success rate.
Crawl any site at scale, without fighting infrastructure.
Crawlbase handles proxies, fingerprints, and CAPTCHAs so your team ships data pipelines instead of maintaining crawl plumbing. Up to 20,000 requests free, no card required.