How to Scrape Cars and Bidsauction listings, bids, and specs

Cars and Bids runs public car auctions, and every listing page is a tidy block of structured facts: the year, make, and model of the vehicle, the current high bid, how much time is left on the auction, where the car sits, and a link into the full detail page. For anyone tracking the enthusiast-car market, that grid of live auctions is one of the cleanest public signals around, which is why dealers, researchers, and analysts watch it for pricing trends, demand, and which models are moving.

This guide shows you how to scrape Cars and Bids auction listings with Python. You build a small, runnable scraper that fetches a listing page through the Crawling API, parses a clean record for each auction, handles pagination, and exports the results to JSON and CSV. The whole walkthrough stays scoped to public auction data: the titles, bids, time remaining, and locations anyone can see on a listing page without logging in.

What you will build

A Python script that takes a Cars and Bids listing URL, retrieves the rendered page through the Crawling API, and extracts a structured record per auction card. We use a make-filtered search page as the running example, the same approach the legacy walkthrough used, and pull these fields from each listing:

• Title the auction headline, which carries the year, make, and model of the car.
• Subtitle the short descriptor line under the title (trim level, notable options, reserve status).
• Current bid the highest bid on the auction at the moment of the crawl.
• Time left how much time remains before the auction closes.
• Location the city and region where the car is listed.
• Link the URL to the auction's own detail page.

Why a plain request fails on Cars and Bids

If you point a bare HTTP client at a Cars and Bids listing URL, you rarely get the auctions you came for. Two things work against you. First, the listing grid renders client-side: the site ships a lightweight shell and fills the auction cards in as the page's JavaScript runs, so the initial HTML you receive is often an empty frame with no listings in it. Second, automated traffic gets flagged fast. Datacenter IP ranges and request patterns that do not look like a real browser get met with a challenge or an outright block before you ever reach the auctions.

So a working Cars and Bids scraper needs two things in one request: a browser that renders the page, and an IP that the site reads as a real visitor. You can assemble that yourself with a headless browser and a pool of rotating residential proxies, but keeping that stack healthy is most of the work. The Crawling API folds both into a single call: you send it the listing URL, it renders the page behind a trusted residential IP, handles the rotation and CAPTCHA solving, and returns finished HTML for you to parse.

Prerequisites

You need a few things in place before writing any code. None of them take long.

Basic Python. You should be comfortable writing and running a Python script and installing packages with pip. If you are new to the language, the official Python docs or any beginner course covers the level this tutorial assumes. The walkthrough on how to scrape a website with Python is a gentle starting point if you want one.

Python 3.8 or later. Confirm your version with python --version (or python3 --version). If you do not have it, install it from python.org and make sure Python is on your system PATH.

A Crawlbase account and token. Sign up for a free account, open your dashboard, and copy your token. Because Cars and Bids relies on JavaScript to load its auctions, you want the JavaScript (JS) token rather than the normal one. The free tier includes 1,000 requests with no card, which is plenty to build and test this scraper. Treat the token like a password and keep it out of version control.

Set up the project

Create a virtual environment so project dependencies stay isolated, then install the two libraries the scraper needs. crawlbase is the official client for the Crawling API, and beautifulsoup4 parses the returned HTML so you can pull each field out of the auction cards by CSS selector.

python --version

python -m venv carsandbids-scraper
source carsandbids-scraper/bin/activate

pip install crawlbase beautifulsoup4

On Windows, activate the environment with carsandbids-scraper\Scripts\activate instead of the source line. With both libraries installed, create the script file the rest of the guide builds up:

touch carsandbids_scraper.py

Understanding the listing page

A Cars and Bids search lives at a stable URL. A make-filtered list, for example, is https://carsandbids.com/search/bmw, and the same pattern holds for other makes. The page lays out a grid of auction cards, one per car, and each card carries the same handful of fields: the auction title (year, make, model), a subtitle, a thumbnail, the current bid, the time remaining, the location, and a link into the auction's own page.

Before writing selectors, open a listing page in your browser, right-click an auction card, and choose Inspect. Each auction sits in an li element marked with the auction-item class. Inside it, the title lives in a div.auction-title, the subtitle in a p.auction-subtitle, the location in a p.auction-loc, the thumbnail in an img, and the link in the card's anchor. Those are the elements you target.

Step 1: Fetch the rendered listing page

Start by getting the finished page. Import the CrawlingAPI class, initialize it with your JS token, set the listing URL, and request it. Checking the status code before you parse keeps failures loud instead of silent.

from crawlbase import CrawlingAPI

crawling_api = CrawlingAPI({"token": "YOUR_CRAWLBASE_TOKEN"})

def make_crawlbase_request(url, options):
    response = crawling_api.get(url, options)
    if response["headers"]["pc_status"] == "200":
        return response["body"].decode("utf-8")
    print(f"Failed to fetch the page. Crawlbase status: {response['headers']['pc_status']}")
    return None

if __name__ == "__main__":
    listing_url = "https://carsandbids.com/search/bmw"
    options = {"ajax_wait": "true", "page_wait": 10000}
    html = make_crawlbase_request(listing_url, options)
    print(html[:500] if html else "No HTML returned")

The two options matter for a grid that fills in after load. ajax_wait tells the API to wait for asynchronous content to finish, and page_wait holds for a fixed number of milliseconds (10,000 here) so the late-rendering auction cards appear before the page is captured. The status check reads the pc_status header the Crawling API returns, and a value of 200 means the render succeeded. Run the script and you should see real listing markup, not an empty shell. That confirms rendering works before you write a single selector.

Step 2: Parse the auction cards with BeautifulSoup

With rendered HTML in hand, load it into BeautifulSoup, find every auction card, and pull each field by its selector. Each card is an li.auction-item; the title, subtitle, location, current bid, time left, thumbnail, and link all sit inside it. Guarding each lookup with a presence check keeps extraction resilient when a field is missing, which happens since not every card shows the same data.

from bs4 import BeautifulSoup

BASE = "https://www.carsandbids.com"

def text_of(listing, tag, css_class):
    el = listing.find(tag, class_=css_class)
    return el.text.strip() if el else None

def scrape_listing_page(html_content):
    soup = BeautifulSoup(html_content, "html.parser")
    car_listings = soup.find_all("li", class_="auction-item")

    extracted_data = []
    for listing in car_listings:
        link_tag = listing.find("a")
        thumbnail = listing.find("img")
        extracted_data.append({
            "title": text_of(listing, "div", "auction-title"),
            "sub_title": text_of(listing, "p", "auction-subtitle"),
            "current_bid": text_of(listing, "span", "bid-value"),
            "time_left": text_of(listing, "span", "td-time"),
            "location": text_of(listing, "p", "auction-loc"),
            "thumbnail": thumbnail["src"] if thumbnail else None,
            "link": BASE + link_tag["href"] if link_tag else None,
        })
    return extracted_data

The text_of helper queries one element inside a card and returns None when it is missing, instead of throwing on a .text call against nothing. The title comes from div.auction-title and carries the year, make, and model; the subtitle from p.auction-subtitle; the location from p.auction-loc; the current bid from the bid value span; and the time remaining from the time-display span. The link is built by prefixing the card's relative href with the site base so you store an absolute URL.

Step 3: Assemble the script and export JSON and CSV

Now wire the fetch and the parse into one runnable script, then write the records to both JSON and CSV so you can load them into a notebook or a spreadsheet. Fetch the rendered listing page, hand it to the parser, and dump the structured rows.

import csv
import json
from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup

crawling_api = CrawlingAPI({"token": "YOUR_CRAWLBASE_TOKEN"})
BASE = "https://www.carsandbids.com"
FIELDS = ["title", "sub_title", "current_bid", "time_left", "location", "thumbnail", "link"]

def make_crawlbase_request(url, options):
    response = crawling_api.get(url, options)
    if response["headers"]["pc_status"] == "200":
        return response["body"].decode("utf-8")
    print(f"Failed to fetch the page. Crawlbase status: {response['headers']['pc_status']}")
    return None

def text_of(listing, tag, css_class):
    el = listing.find(tag, class_=css_class)
    return el.text.strip() if el else None

def scrape_listing_page(html_content):
    soup = BeautifulSoup(html_content, "html.parser")
    car_listings = soup.find_all("li", class_="auction-item")

    extracted_data = []
    for listing in car_listings:
        link_tag = listing.find("a")
        thumbnail = listing.find("img")
        extracted_data.append({
            "title": text_of(listing, "div", "auction-title"),
            "sub_title": text_of(listing, "p", "auction-subtitle"),
            "current_bid": text_of(listing, "span", "bid-value"),
            "time_left": text_of(listing, "span", "td-time"),
            "location": text_of(listing, "p", "auction-loc"),
            "thumbnail": thumbnail["src"] if thumbnail else None,
            "link": BASE + link_tag["href"] if link_tag else None,
        })
    return extracted_data

def export(rows, name="carsandbids_listings"):
    with open(f"{name}.json", "w", encoding="utf-8") as f:
        json.dump(rows, f, indent=2, ensure_ascii=False)
    with open(f"{name}.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=FIELDS)
        writer.writeheader()
        writer.writerows(rows)
    print(f"Saved {len(rows)} auctions to {name}.json and {name}.csv")

def main():
    url = "https://carsandbids.com/search/bmw"
    options = {"ajax_wait": "true", "page_wait": 10000}
    html = make_crawlbase_request(url, options)
    if not html:
        return
    rows = scrape_listing_page(html)
    export(rows)

if __name__ == "__main__":
    main()

Run the full script with python carsandbids_scraper.py. It fetches the rendered listing page, parses one row per auction, and writes both carsandbids_listings.json and carsandbids_listings.csv. The shared FIELDS list keeps the CSV column order in step with the dictionary keys, so the two exports never drift apart.

What the output looks like

You get a clean list of auction records, in page order, ready to write to JSON, CSV, or a database.

[
  {
    "title": "2014 BMW 335i Sedan",
    "sub_title": "No Reserve: Turbo 6-Cylinder, M Sport Package, California-Owned",
    "current_bid": "$9,500",
    "time_left": "2 days",
    "location": "Los Angeles, CA 90068",
    "thumbnail": "https://media.carsandbids.com/cdn-cgi/image/width=768/photos/rkVPlNqQ.jpg",
    "link": "https://www.carsandbids.com/auctions/9QxJ8nV7/2014-bmw-335i-sedan"
  },
  {
    "title": "2009 BMW 328i Sports Wagon",
    "sub_title": "No Reserve: Inspected 3.0-Liter 6-Cylinder, Premium Package",
    "current_bid": "$12,750",
    "time_left": "5 hours",
    "location": "San Diego, CA 92120",
    "thumbnail": "https://media.carsandbids.com/cdn-cgi/image/width=768/photos/3g6kOmG9.jpg",
    "link": "https://www.carsandbids.com/auctions/30n7Yqaj/2009-bmw-328i-sports-wagon"
  }
]

Handling pagination

One search page is a demo; a real research job runs across every page of results. Cars and Bids paginates its search with a ?page= parameter, so you can walk the pages by incrementing the number until a page returns no auction cards, then stop. Pace the requests with a short delay so you are not hammering the site in a tight loop.

import time

def scrape_all_pages(search_url, max_pages=10):
    options = {"ajax_wait": "true", "page_wait": 10000}
    all_rows = []
    for page in range(1, max_pages + 1):
        page_url = f"{search_url}?page={page}"
        html = make_crawlbase_request(page_url, options)
        if not html:
            break
        found = scrape_listing_page(html)
        if not found:
            print(f"No auctions on page {page}; stopping.")
            break
        all_rows.extend(found)
        print(f"Page {page}: {len(found)} auctions")
        time.sleep(2)
    return all_rows

The empty-results break stops you early when the search runs out of pages, and the time.sleep(2) between requests paces the run so you are not flagged for rapid-fire traffic. Swap the single fetch in main for a call to scrape_all_pages("https://carsandbids.com/search/bmw") and the rest of the pipeline (parse, export) carries the combined list straight through. To track an auction over time, run the job on a schedule and stamp each export with the date, then diff successive snapshots to see how bids and time remaining moved.

Staying unblocked

Even with rendering handled, the site watches for scraper-shaped traffic. A few habits keep a run healthy, and they apply to any hard target.

• Pace your requests. Spread requests out with a delay between pages rather than crawling everything at full speed, and schedule heavier jobs during off-peak hours to ease load on the site.
• Lean on rotation. A pool of residential IPs spreads requests across many real-user addresses so no single one trips a rate limit. The Crawling API handles this for you; if you roll your own stack, this is the part to get right.
• Retain only what you need. Store the auction fields your project uses and discard the rest, and re-check your selectors periodically so the scraper keeps pace with markup changes.

For the broader playbook on avoiding blocks, see how to scrape websites without getting blocked, and for more on why rendering matters here, how to crawl JavaScript websites. If you are pulling this data for pricing work, the guide on web scraping for price intelligence covers turning raw listings into a usable signal.

Is it legal to scrape Cars and Bids?

Whether scraping Cars and Bids is allowed depends on the site's Terms of Service, your jurisdiction, and what you do with the data. The site's terms govern automated access, so scraping can run against those terms regardless of how careful your tooling is. None of the code here changes that; it just makes the technical part work. Read the Cars and Bids Terms of Service and its robots.txt, and treat both as the boundary for what you collect. For commercial or competitive use, the legal picture gets more complex, and consulting a legal expert about your specific case is the sensible move.

A few lines worth holding to. Collect only public auction data: the titles, subtitles, current bids, time remaining, locations, and listing links that anyone can see on a search page without an account. Keep your request volume low enough that you are not straining the site's servers, and avoid personal data, including anything tied to identifiable sellers, bidders, or commenters beyond what is publicly listed. Do not redistribute the listing photos or descriptions as your own, since that media is copyrighted.

This guide is deliberately scoped to public listing pages because that is the line that keeps the work defensible. It does not cover anything behind a login, account or bidding data, personal information, or any attempt to bypass authentication or a CAPTCHA you are not entitled to pass. If your project needs more than public listing data, the right path is an official data agreement with the site rather than a cleverer scraper.

Frequently Asked Questions (FAQs)

Why does a plain request return no auctions from Cars and Bids?

The listing grid renders client-side: the site ships a near-empty shell and fills the auction cards in as its JavaScript runs, so a raw request often gets a frame with no listings in it. On top of that, the site challenges or blocks traffic that does not look like a real browser. Rendering the page through the Crawling API behind a trusted IP, with ajax_wait and page_wait set, solves both, which is why the scraper here routes its request through it.

Which Crawlbase token should I use for Cars and Bids?

Use the JavaScript (JS) token. Cars and Bids loads its auctions dynamically, so it needs the rendering that the JS token enables; the normal token returns the unrendered shell. The free tier includes 1,000 requests with no card, which is enough to build and test the scraper.

How do I scrape a specific make or search on Cars and Bids?

Point the scraper at the search URL you want. A make filter is just a path, for example https://carsandbids.com/search/bmw for BMW, so swapping the make at the end of the URL targets a different set of auctions. To cover the full result set, walk the ?page= parameter until a page returns no cards.

What fields can I extract from a Cars and Bids listing?

From each auction card you can pull the title (year, make, model), the subtitle, the current bid, the time remaining, the location, the thumbnail image, and the link to the auction's detail page. The parser maps each of these by its CSS selector, and you can drop the fields you do not need from the output dictionary.

How do I handle pagination across many auctions?

Cars and Bids paginates its search with a ?page= parameter. Loop the page number, fetch and parse each page, and break when a page returns no auction cards. Add a short delay between requests so you pace the run rather than firing pages back to back, and collect the rows into one list before exporting.

How do I avoid getting blocked while scraping Cars and Bids?

Keep your per-IP request rate low, add a delay between pages, and route through rotating residential IPs so no single address trips a rate limit. The Crawling API manages rotation, a trusted IP pool, and CAPTCHA handling for you; if you build your own stack, that is the part to invest in. Watch the pc_status values and back off when you start seeing challenges.

HR

Hassan Rehan

Software Engineer · Crawlbase

Software engineer at Crawlbase writing hands-on guides on rotating proxies, scraping, and the practical details of wiring proxies into real code.