Crawlbase’s Frequently Asked Questions
Top frequently asked questions
Do I need to know how to code to use the Crawling API?
At the moment the only way to use Crawlbase is by integrating it to your product or service, therefore you will need to know development or know/pay someone which can help you.
Please note that we have some very easy to integrate libraries.
Where can I get the API keys?
You can get the API keys or request tokens from the Crawlbase Account Documentation page.
Can the 30 URLs-per-second limit be increased for large-scale crawls?
The 30 URLs-per-second limit applies to LinkedIn crawls. For other websites, we can evaluate and potentially increase the limit on a case-by-case basis. Please contact us to discuss your specific needs.
Do you sell a list of proxies?
We want to offer more than just a list or lists of proxies, therefore our API and Crawler products are built on top of thousands of residential and datacenter worldwide proxies combined with artificial intelligence plus our engineering team which helps delivering the best data results possible. If you are looking just for proxies and you don't want to get any of the other benefits, we recommend that you use the Smart Proxy
What is the difference between Crawling API and Scraper API?
The Crawling API is a generic API that can be used to crawl any website and get the raw source page. It also gives the option to use our classified scrapers. The Scraper API is an API that focuses on auto scraping and parsing HTML pages into structured data. One of their differences as well, the Crawling API is tiered pricing and payments are made at the end of every month based on the request consumption while the Scraper API is subscription-based.
Can the Smart Proxy be used with HTTPS?
Yes, the Smart Proxy can be used for HTTPS websites, but the connection to the proxy must be done with http. So when connecting to the proxy use
http://smartproxy.crawlbase.com and not
All HTTPS traffic that you send to any website using the Smart Proxy, will continue to be securely transfered and we won't be able to access it in any way.
Is it possible to change my account email address?
Yes, you can change your account email address. To do so, visit the Account page and look for the "Change email address" option. After updating your email address, you will receive a confirmation email at the new address. Please confirm the change by following the instructions in the email.
Need help? Contact us
Please contact us for any type of query regarding products
Start crawling and scraping the web today
Try it free. No credit card required. Instant set-up.