Ecommerce Web ScrapingPrices, Catalogs, and Staying Unblocked

Scraping an e-commerce site is two problems wearing one name. The first is extraction: pulling the price, the title, the stock state, and the reviews out of a product page reliably enough that a number you store today still means the same thing next week. The second is access: getting that page at all, at volume, from a target whose whole business depends on telling your scraper apart from a buyer. Most guides only solve the first and then act surprised when the second one returns a wall of 403s.

This post takes both seriously. It covers the handful of data points on a retail page that are actually worth collecting, the anti-bot reality you hit the moment you go past a few hundred requests, how to choose the proxy or fetch type that matches a given store's defenses, a runnable extraction example, and the point where rolling your own stack stops paying off. The goal is a price tracker or catalog feed that still works in a month, not a script that runs once on your laptop and dies in production.

What data on a product page is worth scraping

A retail catalog is wide, but the fields that drive real decisions are few. Collect these and you have covered most of why anyone scrapes a store in the first place.

• Price. The single most-tracked field. Useful only when it is clean: strip the currency symbol, normalize the decimal separator, and store the currency code separately so a price is a number you can compare, not a string you have to re-parse later.
• Availability and stock. "In stock", "2 left", "ships in 3 weeks", and "unavailable" are different signals. Capture the raw state and a normalized boolean, because a competitor going out of stock is often more actionable than a price change.
• Catalog structure. Title, brand, category path, SKU or product ID, and variant axes (size, color). The product ID is what lets you match the same item across sites and across time, so treat it as the primary key.
• Reviews and ratings. Average score, review count, and review text. Counts and averages move slowly and are cheap to track; full review text is heavier and usually paginated behind JavaScript.
• Media and copy. Image URLs and the description, when you are comparing how listings are merchandised rather than just priced.

The discipline that matters here is normalization at capture time. A field you store raw and "clean later" never gets cleaned. Decide the shape of each field before you write the parser, and reject rows that do not fit rather than silently storing a malformed price.

The anti-bot reality on retail sites

Public catalog pages look open, and a single curl against one usually works. That early success is misleading. Retail sites are among the most defended targets on the web, because scraping competitors is something their own teams do, so they know exactly what to watch for. The defenses escalate roughly in this order.

Rate limiting. The simplest layer. Too many requests from one IP in a short window and you get throttled or temporarily banned. This is what catches naive scrapers first, and it is the easiest to defeat by spreading requests across many IPs.

IP reputation. The site checks whether your address belongs to a hosting provider rather than a home connection. Datacenter ranges sit in known ASNs, so a single lookup flags them. A tolerant store ignores this; a hardened one blocks datacenter traffic on sight, which is the whole reason the proxy datacenter vs residential proxies decision exists.

Fingerprinting. Beyond the IP, the site inspects your TLS handshake, header order, and JavaScript environment. A request that claims to be Chrome but does not behave like a browser stands out even from a perfect residential IP. This is where header-only scrapers quietly start getting served decoy pages or challenges.

JavaScript-rendered content. Many modern storefronts ship a near-empty HTML shell and build the price, stock, and reviews client-side. Fetch the raw HTML and the fields you want are not there. You need a real browser to execute the page, or an endpoint that does it for you.

Active challenges. CAPTCHAs and managed bot-detection services that score every request and gate suspicious ones. By the time you hit these, header tweaks are not enough, and the realistic options are convincing real-user traffic or a managed fetch that handles the challenge server-side.

Choosing the proxy or fetch type per target

There is no single right setup for "e-commerce," because retail sites sit all across the defense spectrum. The decision is which type of access matches the store in front of you. A proxy is one layer of indirection that makes the request from a different IP; the question is which kind of IP, and whether you also need a browser.

Two axes are at play. The first is IP trust: datacenter is fast and cheap but obvious, residential reads as a real household, and static residential (ISP) adds a stable address that survives a logged-in session without rotating out mid-request. The full middle-ground tradeoff is in ISP vs residential proxies. The second axis is rendering: if the price only appears after JavaScript runs, no IP choice alone fixes that, you need a browser in the loop. Rotation across a pool is what keeps any single address under the rate limit, and a managed gateway gives you that without maintaining IP lists; see how to use rotating proxies for the mechanics.

Start at the cheap end. Run the target through a rotating datacenter pool with plain HTML first. If you get blocks or empty fields, that failure tells you exactly which rung to climb to next.

A practical extraction example

Here is the shape of a real extractor: fetch the page, parse the fields, normalize them, and reject anything malformed. This uses Python with requests and a parser; swap the selectors for your target's actual markup.

import re
import requests
from bs4 import BeautifulSoup

def parse_product(html):
    soup = BeautifulSoup(html, "html.parser")
    raw_price = soup.select_one(".price").get_text(strip=True)
    # Normalize at capture: strip symbols, keep a real number
    price = float(re.sub(r"[^\d.]", "", raw_price))
    stock = soup.select_one(".stock").get_text(strip=True)
    return {
        "sku": soup.select_one("[data-sku]")["data-sku"],
        "title": soup.select_one("h1").get_text(strip=True),
        "price": price,
        "in_stock": "out" not in stock.lower(),
    }

# Plain fetch works only on tolerant, static targets
resp = requests.get("https://example.com/product/123")
print(parse_product(resp.text))

That works on a tolerant, static store and breaks the moment the target either blocks your IP or renders the price in JavaScript. When resp.text comes back as a block page or a shell without the price, you do not rewrite the parser, you change how you fetch. Route the request through a managed endpoint that handles the IP rotation and runs the page in a browser, and the same parser runs against real DOM.

# Same parser, different fetch: the API rotates IPs and
# renders the page server-side, then returns the DOM.
resp = requests.get(
    "https://api.crawlbase.com/",
    params={
        "token": "_YOUR_TOKEN_",
        "url": "https://example.com/product/123",
        "javascript": "true",
    },
)
print(parse_product(resp.text))

The lesson is the separation: extraction logic is your code and rarely changes, while access is a knob you turn per target. Keep them apart and a store hardening its defenses costs you a config change, not a rewrite. If you would rather not write the parser at all on common marketplaces, a structured data endpoint returns parsed JSON fields directly, trading flexibility for not maintaining selectors.

What to do with the data

The extraction is the means; the e-commerce decisions are the point. The legacy version of this topic sprawled into nine marketing tactics, but the durable uses come down to a few.

Price monitoring. Track competitor prices over time and you see not just the current number but the pattern: when a rival discounts, how deep, how often. That is the difference between reacting to a price drop and anticipating it. Tracking a handful of competitors across hundreds of SKUs by hand is impossible; a scheduled scrape does it in minutes.

Stock and assortment tracking. Knowing what a competitor carries, and when items go out of stock, surfaces gaps you can fill and demand you can meet while they cannot. Out-of-stock signals are often more actionable than price.

Catalog enrichment and matching. Pull richer descriptions, images, and specs to improve your own listings, and use shared product IDs to match the same item across marketplaces for true like-for-like comparison.

Review and sentiment monitoring. Aggregate ratings and review text across products to see what customers praise and complain about, on your listings and competitors', without manually reading thousands of reviews.

When a managed crawling API pays off

The build-versus-buy line is real, and it is honest to name where each side wins. Roll your own when targets are tolerant, mostly static HTML, and scraping is the core thing your product does rather than a supporting feed. In that case a plain rotating proxy plus your own parser is the lean, correct choice, and a managed gateway still spares you the IP-list busywork.

Buy when the targets fight back. Once you are maintaining a headless browser fleet, rotating residential IPs, fingerprint logic, and retry-on-block handling, you have rebuilt a crawling API by hand, usually at higher cost and lower reliability. A managed endpoint absorbs rotation, rendering, retries, and challenge handling behind one request, so your code shrinks to "send a URL, parse the result." The deeper comparison of owning the IPs versus offloading the job is in backconnect proxy vs crawling API.

Frequently Asked Questions (FAQs)

Is it legal to scrape e-commerce websites?

Scraping publicly accessible data is broadly permitted in many jurisdictions, but legality depends on what you collect and how. Avoid personal data, respect the site's terms and rate limits, do not scrape content behind a login you are not authorized to access, and consult a lawyer for anything commercial. This is general information, not legal advice.

Why does my scraper get blocked on retail sites?

Usually one of three things: too many requests from one IP (rate limiting), an address that reads as a datacenter rather than a real user (IP reputation), or a request that does not behave like a real browser (fingerprinting). The fix is to spread requests across rotating IPs, use residential exits on hardened targets, and send believable browser fingerprints, or hand the whole job to a managed fetch that does all three. See how to scrape without getting blocked.

What proxy type is best for scraping e-commerce sites?

It depends on the store's defenses. Use rotating datacenter proxies for tolerant catalogs with static HTML, residential proxies for sites that block datacenter IPs, and ISP (static residential) when you need to hold a logged-in session for account-specific pricing. Start at the cheap end and escalate only when you actually get blocked.

How do I scrape product prices that load with JavaScript?

A plain HTML fetch will not see them, because the price is built in the browser after the page loads. You need to execute the JavaScript, either by driving a real browser yourself (see web scraping with Python and Selenium) or by using a crawling API with rendering enabled, which runs the page server-side and returns the finished DOM you can parse normally.

Should I build my own scraper or use a crawling API?

Build your own when targets are tolerant and static and scraping is your core product. Use a crawling API when targets fight back, because maintaining residential IPs, a headless browser fleet, fingerprints, and retry logic effectively rebuilds one at higher cost. Both can sit behind the same parser, so the choice is how much of the access stack you want to run yourself.

How often should I scrape competitor prices?

Match the cadence to how fast prices move and what you will do with the data. Daily is plenty for most catalogs; fast-moving categories or flash sales may warrant hourly on a small set of key SKUs. Scraping more often than you act on the data just raises your block risk and cost without adding signal.

BA

Bilal Ahmed

Software Engineer · Crawlbase

Software engineer who wrote some of the most-read pieces on the Crawlbase blog, covering web scraping, proxies, and data tooling.