Web Scraping with Python and SeleniumA Build-Along Guide

Selenium drives a real browser. That is its whole reason to exist for scraping: when a page builds its content with JavaScript after the initial HTML loads, a plain HTTP request hands you an empty shell, but a browser runs the scripts and gives you the finished DOM. The cost is that you are now running Chrome for every page, which is slower and heavier than fetching HTML. This guide builds a working Selenium scraper on current tooling (Selenium 4), then shows where driving a browser yourself stops being worth it.

The version gap matters here. Most Selenium tutorials still teach Selenium 3 patterns: find_element_by_id, manually downloaded ChromeDriver binaries, executable_path arguments. All of that is removed or deprecated. Selenium 4 replaced the find helpers with a single By locator API, and Selenium Manager (built in since 4.6) now resolves the right driver for your installed browser automatically, so the driver-download step most guides open with is gone. Everything below is written for that current setup.

What you need

Three things: Python 3.8 or newer, Google Chrome installed, and the Selenium package. That is it. You do not need to download ChromeDriver, and you do not need webdriver-manager on a current install, because Selenium Manager handles the driver for you.

# Selenium 4.6+ ships Selenium Manager, which resolves
# the matching driver for your installed Chrome.
pip install selenium

Launch headless Chrome

Selenium 4 configures the browser through an Options object passed to the driver. For scraping you almost always want headless mode (no visible window), plus a real window size and a user agent, because some sites behave differently when the viewport is tiny or the agent screams automation.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")
options.add_argument("--window-size=1920,1080")
options.add_argument("--disable-gpu")
options.add_argument(
    "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36"
)

# No driver path needed. Selenium Manager resolves it.
driver = webdriver.Chrome(options=options)

If you do need to pin a specific driver binary, that goes through a Service object now (webdriver.Chrome(service=Service("/path/to/chromedriver"), options=options)), not the removed executable_path keyword. For most people, leaving Selenium Manager to do its job is the right call.

Open a page and locate elements

With a driver in hand, driver.get(url) navigates and blocks until the initial load fires. Then you find elements with the By API. This is the single biggest change from Selenium 3: every find_element_by_* helper is gone, replaced by find_element(By.X, value).

from selenium.webdriver.common.by import By

driver.get("https://quotes.toscrape.com/")

# One element, then many.
title = driver.find_element(By.TAG_NAME, "h1").text
quotes = driver.find_elements(By.CLASS_NAME, "quote")

for q in quotes:
    text = q.find_element(By.CLASS_NAME, "text").text
    author = q.find_element(By.CLASS_NAME, "author").text
    print(author, "-", text)

The locators worth knowing are By.ID, By.CLASS_NAME, By.CSS_SELECTOR, and By.XPATH. CSS selectors cover most needs and read cleanly; reach for XPath only when you need to match on text content or traverse upward to a parent, which CSS cannot do. Note that find_element raises if nothing matches, while find_elements returns an empty list, so loop over the plural form and check the length rather than wrapping single lookups in try/except.

Wait for dynamic content

Here is the mistake that breaks most beginner Selenium scrapers: calling find_element the instant after get() returns. On a JavaScript-rendered page the element you want may not exist yet, and you get a NoSuchElementException on a page that loads fine when you watch it. The fix is an explicit wait, which polls until a condition is true or a timeout expires.

Do not reach for time.sleep(). A fixed sleep either wastes time when the page is fast or fails when it is slow; an explicit wait returns the moment the element is ready and only times out if it genuinely never appears.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

# Wait until at least one .quote is present in the DOM.
wait.until(
    EC.presence_of_element_located((By.CLASS_NAME, "quote"))
)

# Now it is safe to read the rendered content.
quotes = driver.find_elements(By.CLASS_NAME, "quote")

Use presence_of_element_located when you only need the element in the DOM, and visibility_of_element_located or element_to_be_clickable when you are about to read text or click. The locator is passed as a tuple, which is an easy thing to get wrong: it is ((By.CLASS_NAME, "quote")), one tuple argument, not two.

Handle pagination

Most real targets spread data across pages. The pattern is a loop: scrape the current page, find the next-page control, click it, wait for the new content, repeat until the control is gone. The trap is that the page object goes stale after navigation, so you re-find the next button each iteration rather than holding a reference to it.

from selenium.common.exceptions import NoSuchElementException

all_quotes = []

while True:
    wait.until(
        EC.presence_of_element_located((By.CLASS_NAME, "quote"))
    )
    for q in driver.find_elements(By.CLASS_NAME, "quote"):
        all_quotes.append(q.find_element(By.CLASS_NAME, "text").text)

    try:
        next_btn = driver.find_element(By.CSS_SELECTOR, "li.next a")
    except NoSuchElementException:
        break  # last page reached
    next_btn.click()

print("scraped", len(all_quotes), "quotes")
driver.quit()

Always call driver.quit() when you are done. It closes the browser and the driver process; driver.close() only closes the current window and leaves the process running, which leaks Chrome instances fast in a loop. For infinite-scroll pages instead of a next button, the equivalent is scrolling with driver.execute_script("window.scrollTo(0, document.body.scrollHeight)") in a loop and waiting for the item count to stop growing.

Route Selenium through a proxy

Scrape any target at volume from one IP and you will get rate-limited or blocked. Routing the browser through a proxy server spreads requests across different addresses so no single one trips a limit. The simplest form passes the proxy as a Chrome argument.

options = Options()
options.add_argument("--headless=new")
options.add_argument("--proxy-server=http://proxy.example.com:8080")

driver = webdriver.Chrome(options=options)
driver.get("https://httpbin.org/ip")
print(driver.find_element(By.TAG_NAME, "body").text)

That works for an unauthenticated proxy. The catch: Chrome's --proxy-server flag does not accept username and password in the URL, so a credentialed proxy needs a browser extension that injects the auth header, or a local relay that holds the credentials. This is one of the rougher edges of driving Chrome directly. A rotating endpoint that authenticates by token sidesteps it: you point Selenium at one host and the rotation and trust live behind it. The tradeoffs between managing your own pool and using a managed endpoint are covered in backconnect proxy vs crawling API, and if your targets are hardened, datacenter vs residential proxies explains which IP type you actually need.

When a browser is the wrong tool

Selenium is right when the page genuinely needs a browser: heavy client-side rendering, interaction-gated content, or workflows that depend on clicking and typing. It is the wrong tool when you reach for it out of habit. If the data is in the initial HTML, requests plus a parser is an order of magnitude faster and lighter. And once a target fights back with serious anti-bot defenses, a raw headless browser gets fingerprinted and blocked anyway, and you end up rebuilding rotation, retries, and stealth by hand.

The third row is where most production scraping lands. A managed crawling API renders the page server-side, rotates IPs, retries on blocks, and hands you finished HTML from a single request, so you never run or fingerprint-harden a browser fleet yourself. You keep Selenium for genuine interaction work and offload the high-volume, gets-blocked work to the API.

Calling it is a single HTTP request, no driver, no waits, no proxy auth dance: send your token and the target URL, ask for rendering, and read the body back.

import requests

resp = requests.get(
    "https://api.crawlbase.com/",
    params={
        "token": "_YOUR_TOKEN_",
        "url": "https://quotes.toscrape.com/js/",
        "javascript": "true",  # render the page server-side
    },
)
print(resp.status_code)
print(resp.text[:500])

Same rendered result you would get from Selenium, without the browser, the explicit waits, or the proxy-auth workaround. For a deeper comparison of running your own pool versus an endpoint that owns the whole job, see the best proxies for web scrapers.

Frequently Asked Questions (FAQs)

Do I still need to download ChromeDriver for Selenium 4?

No. Since Selenium 4.6, the built-in Selenium Manager detects your installed Chrome and downloads the matching driver automatically, so a plain pip install selenium is enough. You only specify a driver path through a Service object when you deliberately need to pin a particular binary; the old executable_path keyword has been removed.

Why does find_element throw NoSuchElementException on a page that loads fine?

Because the element is rendered by JavaScript after the initial load, and your code queried the DOM before it existed. Replace the immediate lookup with an explicit wait: WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "quote"))). The wait polls until the element appears or the timeout expires, which avoids both fixed sleeps and race conditions.

Should I use Selenium or BeautifulSoup for web scraping?

Use requests with a parser like BeautifulSoup when the data is present in the page's initial HTML; it is far faster and lighter because it never starts a browser. Use Selenium when the content is rendered by JavaScript or you need to click, type, scroll, or log in. Many scrapers combine both: Selenium renders the page, then BeautifulSoup parses the resulting HTML.

How do I use an authenticated proxy with Selenium?

Chrome's --proxy-server argument does not accept a username and password in the URL, so a credentialed proxy needs a browser extension that injects the auth header or a local relay that holds the credentials. A token-authenticated rotating endpoint avoids the problem: you point Selenium at one host and the credentials live behind it rather than in the launch flags.

Is Selenium good for scraping at scale?

It scales poorly. Each page runs a full browser, which is slow and memory-hungry, and on hardened targets a raw headless browser still gets fingerprinted and blocked. For high volume, a managed crawling API that renders server-side, rotates IPs, and retries on blocks is usually the better fit, leaving Selenium for genuine interaction-heavy work.

What is the difference between driver.close() and driver.quit()?

driver.close() closes the current browser window but leaves the driver process and any other windows running. driver.quit() closes every window and shuts down the driver process. In a scraping loop, always finish with driver.quit(), otherwise orphaned Chrome processes pile up and exhaust memory.

BA

Bilal Ahmed

Software Engineer · Crawlbase

Software engineer who wrote some of the most-read pieces on the Crawlbase blog, covering web scraping, proxies, and data tooling.

FQ

Farah Qadeer

Content Visualization · Crawlbase

Content visualization specialist at Crawlbase, turning dense proxy and web-scraping topics into clear visuals and build-along guides.