How to Use Rotating ProxiesPer-Request vs Sticky Sessions, in Code

A rotating proxy is one host and port that hands you a different exit IP over time, so a scraper that sends a thousand requests does not send them all from one address the target can rate-limit and block. You point your HTTP client at the rotating endpoint instead of the site, and the rotation happens behind it. That is the whole mechanism. Everything else (per-request vs sticky, residential vs datacenter, build vs buy) is a choice layered on top of it.

This guide is the practical version. It covers the one decision that actually changes your code, rotate a fresh IP on every request or hold one IP across a session, shows a working Python example for each, and then says plainly when pointing at a managed rotating endpoint beats assembling and babysitting your own pool. If you want the background on what a proxy is at all, what is a proxy server covers the layer of indirection this all sits on.

The one decision that matters: per-request vs sticky session

Almost every rotating-proxy question reduces to a single choice, and it is not which provider. It is whether you want a new IP on every request or the same IP held across a sequence of requests. The two modes solve opposite problems, and picking the wrong one is the most common reason a scrape that "uses rotating proxies" still gets blocked.

The logic is straightforward. When each request is independent (you are pulling one product page after another and nothing carries over), a fresh IP per request spreads the load so no single address trips a rate limit. But the moment requests depend on each other, a login that sets a cookie, a cart you add to and then check out, rotating mid-flow breaks you. The site sees your session jump from one country to another between two clicks and drops you. There you want a sticky session: the same exit IP for the whole sequence, then a new one for the next user.

Per-request rotation in Python

The cleanest way to rotate per request is to let a managed endpoint do it: you point your client at one host and the exit IP changes on the back end. To your code it is just a proxy, so nothing about requests changes except the proxy URL. Here is a minimal scrape that proves the IP is moving by hitting an echo service a few times.

# Per-request rotation: one endpoint, a new exit IP each call.
import requests

token = "_YOUR_TOKEN_"
endpoint = f"http://{token}:@smartproxy.crawlbase.com:8012"
proxies = {"http": endpoint, "https": endpoint}

# Hit an echo service 3 times; each line should show a different IP.
for _ in range(3):
    resp = requests.get("https://httpbin.org/ip", proxies=proxies, verify=False)
    print(resp.json()["origin"])

Swap httpbin.org/ip for your real target and the same proxy config carries every request through a rotating exit. The verify=False flag skips TLS verification because traffic is relayed through the proxy's tunnel; in production you would point requests at the provider's CA bundle instead of disabling verification. From here, parsing is ordinary BeautifulSoup or whatever you already use, the proxy is invisible to it.

Sticky sessions in Python

For a flow that must look like one user, you keep the exit IP fixed across the requests that belong together. With a managed endpoint, the usual mechanism is to use a single requests.Session (so cookies persist) and pass a session identifier the gateway uses to pin one IP for a window. The session is what carries the login cookie from request to request; the pinned IP is what stops the site from seeing your "user" teleport mid-flow.

# Sticky session: one IP + one cookie jar across a multi-step flow.
import requests

token = "_YOUR_TOKEN_"
endpoint = f"http://{token}:@smartproxy.crawlbase.com:8012"

session = requests.Session()  # persists cookies across requests
session.proxies = {"http": endpoint, "https": endpoint}

# Step 1: log in (sets a cookie). Step 2: read a gated page.
# Same IP + same cookie jar make both look like one visitor.
session.post("https://example.com/login", data={"user": "u", "pass": "p"}, verify=False)
resp = session.get("https://example.com/account/orders", verify=False)
print(resp.status_code)

The exact way you request a sticky window (a header, a session token in the username, or a dedicated port) depends on the provider, so check their docs for the parameter name. The pattern is universal: hold the IP for the requests that share state, release it when the user is done.

Rolling your own rotation

You do not strictly need a managed endpoint to rotate. If you have a list of proxies, you can pick one per request yourself. This is worth understanding because it shows exactly what a managed endpoint is doing for you, and where it breaks down.

# Manual rotation: cycle a list of proxies yourself.
import requests, itertools

pool = [
    "http://user:pass@ip1:port",
    "http://user:pass@ip2:port",
    "http://user:pass@ip3:port",
]
rotation = itertools.cycle(pool)  # round-robin instead of random

def fetch(url):
    proxy = next(rotation)
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=15)

Round-robin with itertools.cycle is more predictable than random choice because it spreads load evenly instead of hammering whatever the random number generator favors. But notice what this code does not handle: a dead IP in the pool, a proxy that returns a block page with a 200 status, retrying a failed request on a different exit, or pruning IPs that have been flagged. In a real run those cases are most of the work, and they are exactly what a managed rotating endpoint absorbs for you. If you want to see the full pattern for cycling addresses, how to rotate an IP address goes deeper on the mechanics.

When a managed endpoint beats rolling your own

The manual version above is fine for a small, stable list against a tolerant site. It stops being fine the moment any of these is true: the target actively fights bots, you need real-user (residential or mobile) IPs rather than datacenter, your pool is large enough that liveness and health checking become a chore, or you need retries and per-request IP selection without writing that orchestration yourself.

A managed rotating endpoint collapses all of that into one host and port. You point your client at it, and it selects an exit IP, rotates per request or holds a sticky session, and retries on the back end when an IP gets blocked. Your scraping logic does not change, it is still just a proxy URL, but the pool management, health checks, and rotation policy stop being your code. The honest tradeoff: you give up fine-grained control over individual IPs in exchange for not maintaining a fleet. For most scraping that is a good trade, because IP plumbing is not the part of the job that creates value.

There is one rung above even that. When the target also requires a rendered browser, sends CAPTCHA challenges, or needs a believable fingerprint, rotating the IP is only part of the fight, and a backconnect proxy vs a crawling API is the comparison to read. A rotating endpoint hands you a clean IP and steps back; a crawling API owns rotation, rendering, and retries end to end and hands you the finished page.

Common failures and how to read them

Rotating proxies fail in a small number of recognizable ways. Knowing which one you are looking at saves hours of guessing.

You still get blocked. Usually one of two things: you are rotating per request on a flow that needs a sticky session (so the site sees your session jump IPs), or your IP type is wrong for the target. A hardened site will drop datacenter IPs no matter how fast you rotate them; it wants real-user IPs. Match the IP type to the defenses, not the rotation speed, the type tradeoff is laid out in datacenter vs residential proxies.

A 200 that is actually a block. Many sites return a CAPTCHA or "are you human" page with a 200 status, so checking status_code == 200 is not enough. Inspect the body for known block markers (a challenge string, an unexpected redirect, a suspiciously short response) before you trust the response. Treat those as failures and retry on a fresh IP.

Authentication errors. A 407 (or a connection that hangs) almost always means the credentials in the proxy URL are wrong or malformed. Double-check the token and the user:pass@host:port shape. If you are debugging proxy status codes in general, how to solve proxy status error codes maps the common ones to causes.

Slow or timing out. Residential and mobile exits route through consumer connections and are slower than datacenter by design. Set a sensible timeout (the example uses 15s) and a retry, rather than letting a single slow exit stall the run. If a managed endpoint is timing out broadly, that is a provider signal, not a code bug.

Best practices that actually move the needle

Most "best practices" lists are padding. These few genuinely change your block rate. Pace your requests instead of bursting (rotation spreads IPs, but a thousand requests in one second from any pool still looks automated). Rotate your User-Agent alongside the IP, since a fixed UA across thousands of "different" IPs is a giveaway. Match the IP type to the target rather than reflexively buying the most expensive tier. And for production, use a provider with a clear logging policy rather than scraped free proxy lists, which are slow, short-lived, and a real security risk, covered in are proxies safe. If your end goal is broader than rotation alone, how to scrape websites without getting blocked puts these in context.

Frequently Asked Questions (FAQs)

How do I use a rotating proxy in Python?

Point your HTTP client at the rotating endpoint instead of the target. With requests, build a proxies dict ({"http": url, "https": url}) where the URL is your provider's host, port, and credentials, and pass it to every requests.get or requests.post. The exit IP changes on the back end, so your parsing code does not change at all. Test it by requesting an echo service like httpbin.org/ip a few times and confirming the IP differs.

What is the difference between per-request rotation and a sticky session?

Per-request rotation gives you a fresh exit IP on every request, which spreads load and suits stateless, high-volume reads. A sticky session holds one IP across a sequence of requests, which is what you need for logins, carts, and any multi-step flow that must look like a single user. Use per-request for independent reads and sticky whenever requests depend on each other.

Why am I still getting blocked with a rotating proxy?

The two usual causes are rotating per request on a flow that needs a sticky session (so the site sees your session jump IPs mid-flow) and using the wrong IP type. A hardened target drops datacenter IPs regardless of how fast you rotate them. Match the IP type to the defenses, hold a sticky session for stateful flows, and check whether a "200" response is actually a CAPTCHA page.

Should I rotate proxies myself or use a managed endpoint?

Rolling your own works for a small, stable list against a tolerant site. A managed endpoint wins once you need real-user IPs, a large pool, health checking, retries, or per-request IP selection, because it absorbs all of that behind one host and port. Performance and cost ranges vary by target and provider; they are not fixed constants. The trade is giving up IP-level control for not maintaining a fleet.

Do rotating proxies work for scraping behind a login?

Yes, but you must use a sticky session, not per-request rotation. Keep the same exit IP and the same cookie jar across the login and the gated pages that follow, so the site sees one consistent visitor. If you need an IP that stays fixed across many separate runs, that is a static residential (ISP) proxy rather than a rotating one.

How fast can I send requests through a rotating proxy?

Faster than from a single IP, because rotation spreads requests across many addresses, but rotation is not a license to burst. A thousand requests in one second from any pool still reads as automated. Pace your requests, set a timeout and retry, and rotate your User-Agent alongside the IP. Residential and mobile exits are slower than datacenter by design, so budget for that in your timeouts.

HR

Hassan Rehan

Software Engineer · Crawlbase

Software engineer at Crawlbase writing hands-on guides on rotating proxies, scraping, and the practical details of wiring proxies into real code.