Search Results for:

Instead of offering standalone proxy lists, our API and Crawler products are integrated with thousands of residential and datacenter proxies from around the world, enhanced by artificial intelligence and our expert engineering team, to deliver the best possible data results. If you're only interested in proxies without the additional benefits, we suggest using Smart Proxy as an alternative.

We do not disclose the exact number of proxies, as it is constantly changing. Our network experts continuously source new worldwide proxies and replace faulty ones, while our AI adjusts the proxy set size depending on the target website. If you are a proxy provider or have proxies to sell contact us to help expand our infrastructure.

We strive to support all websites effectively. If you experience blocks or captchas, please contact us so we can assist and find a solution for your specific situation.

We do not store any credit card information. Instead, all payment data is directly and securely stored with our payment partner, Paddle, which adheres to global payment regulations to ensure your information remains safe.

To add your VAT details, please go to the Settings and Billing page. When entering your card information, select your country, and then click the VAT number button located in the bottom left corner of the window. Enter your VAT details there. If you have already provided your card information, you can simply re-enter it to add the VAT number.

To change the billing email address, first, go to the Account page and initiate the process of reauthorizing your billing details. When the billing details form appears, you'll see your current email address in the bottom right corner next to a "Not you?" button. Click this button and enter the desired email address for billing. Finally, complete the payment reauthorization process to save the updated billing email.

Yes, you can change your account email address. To do so, visit the Account page and look for the "Change email address" option. After updating your email address, you will receive a confirmation email at the new address. Please confirm the change by following the instructions in the email.

Crawlbase doesn't support any illegal activities. Any account being used for illegal purposes will be automatically suspended and no money will be refunded. If you want to know if the data you are storing or the processes you do with such data are legal in your country, please consult with the appropriate person in your country of business operations as we cannot provide legal advice on each individual use case.

there is no limit. The crawler will send back the full html of the page regardless of its size

When you utilize our API to send requests, we employ a comprehensive approach to safeguard your IP address and ensure anonymity during the crawling process. Our API leverages a pool of millions of proxies, enabling us to avoid blocks and CAPTCHAs while effectively masking your IP with each request. This process ensures that your requests remain anonymous and untraceable by the websites you're crawling.
 

It's essential to highlight that we never engage in IP forwarding, as such practices are not feasible within our API-based crawling system. Rest assured, your IP remains protected and inaccessible to external parties throughout the entire crawling process. Your privacy and security are of utmost importance to us.

 

For more complex websites like G2.com, we can implement a custom solution utilizing trained AI bots, which further enhances our ability to avoid proxy failures and blocks. By utilizing premium residential networks within the US, we distinguish legitimate requests from automated bots effectively, ensuring a smooth and uninterrupted crawling experience. Contact our support team for more details.

At the moment the only way to use Crawlbase is by integrating it to your product or service, therefore you will need to know development or know/pay someone which can help you.

Please note that we have some very easy to integrate libraries.

We want to offer more than just a list or lists of proxies, therefore our API and Crawler products are built on top of thousands of residential and datacenter worldwide proxies combined with artificial intelligence plus our engineering team which helps delivering the best data results possible. If you are looking just for proxies and you don't want to get any of the other benefits, we recommend that you use the Smart Proxy

We don't provide the number of proxies as it's constantly changing.

Our team of network experts are constantly looking for new worldwide proxies and replacing fault ones, plus our artificial inteligence might use a smaller or bigger set of proxies depending on the site you are trying to crawl.

If you are a proxy provider or have proxies to sell, please contact us as we are constantly growing our infrastructure.

Yes they do rotate on every request with a new IP. If you don't want that, you can use sessions, take a look at sessions here.

We try our best to support every website. Please contact us to see how we can help.

No, all the Crawling API calls have unlimited bandwidth.

Yes you can use the free requests for javaScript or normal requests.

Unique Proxies is the amount of unique proxies that we have available for each package. IP rotation is the amount of unique IPs which you will be able to obtain in the rotation period. The rotation period varies depending on our IP needs, it can be as low as 24 hours or as high as 1 week.

Yes, the Smart Proxy can be used for HTTPS websites, but the connection to the proxy must be done with http. So when connecting to the proxy use http://smartproxy.crawlbase.com and not https://smartproxy.crawlbase.com.

All HTTPS traffic that you send to any website using the Smart Proxy, will continue to be securely transfered and we won't be able to access it in any way.

No, the Smart Proxy doesn't have any bandwidth limitation.

The first 5000 requests from the Smart Proxy are completely free of charge.

By default a request is retried 110 times for a period of 48 hours from the first time it starts to be processed.

You will always receive a callback in case of success or failure, please check the pc_status and original_status to know the status code.

If your Crawler callback is down, you are notified by email, your crawlers get paused and your last failed request due to downtime at your endpoint, is set to be retried. Your crawlers get resumed when your endpoint becomes available automatically. Our monitoring system checks your endpoint every minute.

"Waiting" means that your requests are in your crawler queue waiting to be processed. "Concurrent crawlers" are the requests that are being crawled at the same time. Concurrent crawlers gets increased by our system if you have many pages to crawl, we also monitor crawlers and increase or decrease the concurrency depending on the pool. "Sets to be retried" are your requests that failed for any reason, they land in your crawler retry queue and are processed with a retry rate up until maximum 110 retries.

You can get the API keys or request tokens from the Crawlbase Account Documentation page.

https://crawlbase.com/dashboard/account/docs

The 30 URLs-per-second limit applies to LinkedIn crawls. For other websites, we can evaluate and potentially increase the limit on a case-by-case basis. Please contact us to discuss your specific needs.