Dotted square
Clouds server

crawler FAQs

How many times are requests re-tried?
Crawler

By default a request is retried 110 times for a period of 48 hours from the first time it starts to be processed.

Was this helpful?
What happens in the case of a permanent failure?
Crawler

You will always receive a callback in case of success or failure, please check the pc_status and original_status to know the status code.

Was this helpful?
What if my webhook endpoint is down?
Crawler

If your Crawler callback is down, you are notified by email, your crawlers get paused and your last failed request due to downtime at your endpoint, is set to be retried. Your crawlers get resumed when your endpoint becomes available automatically. Our monitoring system checks your endpoint every minute.

Was this helpful?
Live monitor wordings
Crawler

"Waiting" means that your requests are in your crawler queue waiting to be processed. "Concurrent crawlers" are the requests that are being crawled at the same time. Concurrent crawlers gets increased by our system if you have many pages to crawl, we also monitor crawlers and increase or decrease the concurrency depending on the pool. "Sets to be retried" are your requests that failed for any reason, they land in your crawler retry queue and are processed with a retry rate up until maximum 110 retries.

Was this helpful?
Where can I get the API keys?
Crawler

You can get the API keys or request tokens from the Crawlbase Account Documentation page.

https://crawlbase.com/dashboard/account/docs

Was this helpful?
Can the 30 URLs-per-second limit be increased for large-scale crawls?
Crawler

The 30 URLs-per-second limit applies to LinkedIn crawls. For other websites, we can evaluate and potentially increase the limit on a case-by-case basis. Please contact us to discuss your specific needs.

Was this helpful?

Need help? Contact us

Please contact us for any type of query regarding products

Contact UsArrow topJust a message away!
Cloud servers

Start crawling and scraping the web today

Try it free. No credit card required. Instant set-up.

Arrow whiteCrawl product data at scale

Webinar: Advanced Techniques in Data Scraping: Beyond the Basics  - 

Sign up now