crawler FAQs

By default a request is retried 110 times for a period of 48 hours from the first time it starts to be processed.

You will always receive a callback in case of success or failure, please check the pc_status and original_status to know the status code.

If your Crawler callback is down, you are notified by email, your crawlers get paused and your last failed request due to downtime at your endpoint, is set to be retried. Your crawlers get resumed when your endpoint becomes available automatically. Our monitoring system checks your endpoint every minute.

"Waiting" means that your requests are in your crawler queue waiting to be processed. "Concurrent crawlers" are the requests that are being crawled at the same time. Concurrent crawlers gets increased by our system if you have many pages to crawl, we also monitor crawlers and increase or decrease the concurrency depending on the pool. "Sets to be retried" are your requests that failed for any reason, they land in your crawler retry queue and are processed with a retry rate up until maximum 110 retries.

You can get the API keys or request tokens from the Crawlbase Account Documentation page.

https://crawlbase.com/dashboard/account/docs

The 30 URLs-per-second limit applies to LinkedIn crawls. For other websites, we can evaluate and potentially increase the limit on a case-by-case basis. Please contact us to discuss your specific needs.