Dotted squareQuestions & Answers

Crawlbase’s Frequently Asked Questions

Clouds server

Top frequently asked questions

Do I need to know how to code to use the Crawling API?
Crawling API

At the moment the only way to use Crawlbase is by integrating it to your product or service, therefore you will need to know development or know/pay someone which can help you.

Please note that we have some very easy to integrate libraries.

Was this helpful?
Where can I get the API keys?
Crawler

You can get the API keys or request tokens from the Crawlbase Account Documentation page.

https://crawlbase.com/dashboard/account/docs

Was this helpful?
How many times are requests re-tried?
Crawler

By default a request is retried 110 times for a period of 48 hours from the first time it starts to be processed.

Was this helpful?
Do you sell a list of proxies?
Crawling API

We want to offer more than just a list or lists of proxies, therefore our API and Crawler products are built on top of thousands of residential and datacenter worldwide proxies combined with artificial intelligence plus our engineering team which helps delivering the best data results possible. If you are looking just for proxies and you don't want to get any of the other benefits, we recommend that you use the Smart Proxy

Was this helpful?
Can the Smart Proxy be used with HTTPS?
Smart proxy

Yes, the Smart Proxy can be used for HTTPS websites, but the connection to the proxy must be done with http. So when connecting to the proxy use http://smartproxy.crawlbase.com and not https://smartproxy.crawlbase.com.

All HTTPS traffic that you send to any website using the Smart Proxy, will continue to be securely transfered and we won't be able to access it in any way.

Was this helpful?
Can the 30 URLs-per-second limit be increased for large-scale crawls?
Crawler

The 30 URLs-per-second limit applies to LinkedIn crawls. For other websites, we can evaluate and potentially increase the limit on a case-by-case basis. Please contact us to discuss your specific needs.

Was this helpful?
Is it possible to change my account email address?
General

Yes, you can change your account email address. To do so, visit the Account page and look for the "Change email address" option. After updating your email address, you will receive a confirmation email at the new address. Please confirm the change by following the instructions in the email.

Was this helpful?
Can I use the free requests for javaScript?
Crawling API

Yes you can use the free requests for javaScript or normal requests.

Was this helpful?
What happens in the case of a permanent failure?
Crawler

You will always receive a callback in case of success or failure, please check the pc_status and original_status to know the status code.

Was this helpful?
What if my webhook endpoint is down?
Crawler

If your Crawler callback is down, you are notified by email, your crawlers get paused and your last failed request due to downtime at your endpoint, is set to be retried. Your crawlers get resumed when your endpoint becomes available automatically. Our monitoring system checks your endpoint every minute.

Was this helpful?

Need help? Contact us

Please contact us for any type of query regarding products

Contact UsArrow topJust a message away!
Cloud servers

Start crawling and scraping the web today

Try it free. No credit card required. Instant set-up.

Arrow whiteCrawl product data at scale

Webinar: How Public Web Data Can Fuel 💼 Business Growth -

Sign up now