# The Smart Proxy in minutes

If your application is not designed to work with an HTTP/S based API to crawl and scrape websites like the Crawling API, we have designed an intelligent rotating proxy that forwards your requests to the Crawling API. You simply use it as a normal proxy in your application.

All Proxy calls should go to http://smartproxy.crawlbase.com and port 8012 using your access token as a proxy username.

Therefore making your first call is as easy as running the following line in the terminal. Go ahead and try it!

curl -x "http://[email protected]:8012" -k "http://httpbin.org/ip"

# How it works?

When you send a request to the proxy, the proxy will authorize your request using your proxy authorization username, your private access token below. It will then redirect your request to the Crawling API and then return the response to your application. If you require to use the extra features of the Crawling API in this mode, you will need to send the HTTP header crawlbaseAPI-Parameters and send the options you require to use. Check the examples section below for real examples.

Private token


# Important Note

It is important to disable SSL verification when using the smart proxy, otherwise, we will not be able to execute our smart AI integration with your requests. Therefore you should skip verifying certificates.

# Request example

  • curl
  • ruby
  • node
  • php
  • python
  • go
// GET request

curl -x "http://[email protected]:8012" -k "http://httpbin.org/ip"

// GET request with custom UserAgent and returning original headers

// Make use of the API parameters, like store=true, device=mobile, get_cookies=true and etc.

// Example of using a custom user agent and storing a copy of the result in Crawlbase Cloud storage.

curl -H "CrawlbaseAPI-Parameters: get_headers=true&store=true&user_agent=Mozilla%2F5.0%20%28Windows%20NT%2010.0%3B%20Win64%3B%20x64%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Chrome%2F91.0.4472.124%20Safari%2F537.36" \
-x "http://[email protected]:8012" \
-k "http://httpbin.org/headers"

// GET request with headless browser (JavaScript Enabled)

curl -H "CrawlbaseAPI-Parameters: javascript=true" -x "http://[email protected]:8012" \
-k "http://httpbin.org/anything"

// POST form request
curl -H 'Content-Type: application/x-www-form-urlencoded' \
-F 'param=value' \
-x "http://[email protected]:8012" \
-k "http://httpbin.org/anything"

// POST JSON request
curl -H "accept: application/json" \
--data '{"key1":"value1","key2":"value2"}' \
-x "http://[email protected]:8012" \
-k "http://httpbin.org/anything"