How to Scrape Apartments.comrentals into structured rows

Apartments.com is one of the largest rental marketplaces on the web, and its listing pages carry exactly the structured data that powers rent benchmarking, market research, and lead generation in real estate: a property name and address, the monthly rent, beds, baths, square footage, and the amenity list. The problem is that those pages are rendered client-side and the site defends hard against automated traffic, so a plain HTTP request hands you a near-empty shell instead of the listing you came for.

This guide shows you how to scrape Apartments.com with Python the reliable way. You build a small, runnable scraper that fetches a rendered listing through the Crawling API, parses the fields you want with BeautifulSoup, and prints clean structured output. We keep the whole walkthrough scoped to public listing data, and the legality section near the end is not boilerplate, so read it before you point this at any real volume.

What you will build

A Python script that takes a public Apartments.com listing URL, retrieves the rendered HTML through the Crawling API, and extracts a structured record for the property. We will use a single rental listing as the running example and pull these fields:

• Name and address the property name and its street address.
• Rent the monthly rent or rent range shown on the listing.
• Beds the number of bedrooms.
• Baths the number of bathrooms.
• Size the square footage of the unit.
• Amenities the list of features, like air conditioning, parking, or in-unit laundry.

Why a plain fetch fails on Apartments.com

If you request an Apartments.com listing URL with a bare HTTP client, you get a response with status 200 and almost none of the listing data in the body. Two things work against you. First, Apartments.com renders much of its listing content in the browser with JavaScript, so the initial HTML is a thin shell that only fills in after the page's scripts run. Second, the site flags automated traffic quickly: datacenter IPs and request patterns that do not look like a real browser get challenged or served a captcha before they ever reach the rendered content.

So a working Apartments.com scraper needs two things in one request: a browser that actually renders the page, and an IP the platform reads as a real visitor. You can assemble that yourself with a headless browser plus a pool of rotating residential proxies, but stitching those together and keeping them healthy is most of the work. The Crawling API folds both into a single call: you send it the URL with a JavaScript token, it renders the page behind a trusted IP, and it returns finished HTML for you to parse.

Prerequisites

You need a few things in place before writing any code. None of them take long.

Basic Python. You should be comfortable writing and running a Python script and installing packages with pip. If you are new to the language, the official Python docs and any beginner course will get you to the level this tutorial assumes.

Python 3.8 or later. Confirm your version with python --version. If you do not have it, install it from python.org or through a distribution like Anaconda.

A Crawlbase account and JS token. Sign up, open your dashboard, and copy your JavaScript (JS) token from the account docs page. Treat the token like a password: it authenticates your requests, so keep it out of version control.

Set up the project

Create a virtual environment so project dependencies stay isolated, then install the two libraries the scraper needs.

python --version

python -m venv apartments_env
source apartments_env/bin/activate

pip install crawlbase beautifulsoup4

On Windows, activate the environment with apartments_env\Scripts\activate instead of the source line. Two dependencies do the work: crawlbase is the official client for the Crawling API, and beautifulsoup4 parses the returned HTML so you can pull out individual fields by CSS selector. If you have not used the parser before, the BeautifulSoup guide is a good companion to this tutorial.

Step 1: Fetch the rendered listing

Start by getting the finished page. Import the CrawlingAPI class, initialize it with your JS token, and request the listing URL. Checking the status code before you parse keeps failures loud instead of silent.

from crawlbase import CrawlingAPI

api = CrawlingAPI({"token": "YOUR_CRAWLBASE_JS_TOKEN"})

def crawl(page_url):
    options = {"ajax_wait": "true", "page_wait": 5000}
    response = api.get(page_url, options)
    if response["status_code"] == 200:
        return response["body"].decode("utf-8")
    print(f"Request failed: {response['status_code']}")
    return None

if __name__ == "__main__":
    page_url = "https://www.apartments.com/2630-n-hamlin-ave-chicago-il/kvl7tm9/"
    html = crawl(page_url)
    print(html[:500] if html else "No HTML returned")

The two wait options matter for a client-rendered target like this. ajax_wait tells the API to wait for asynchronous content to finish loading, and page_wait holds for a fixed number of milliseconds after load so late-rendering elements appear before the page is captured. Five seconds is a reasonable start; raise it if the listing fields come back empty. Run the script with python scraper.py and you should see real listing markup, not the empty shell a plain fetch returns. That confirms rendering works before you write a single selector.

Step 2: Parse the listing fields with BeautifulSoup

With rendered HTML in hand, load it into BeautifulSoup and pull each field by its selector. Apartments.com lays the core listing details out in a predictable structure, so you can map name, rent, beds, baths, size, and amenities to individual selectors. Wrap the whole extraction in helpers that return None or an empty list when an element is missing, so one absent field does not crash the run.

from bs4 import BeautifulSoup

def text_of(soup, selector):
    el = soup.select_one(selector)
    return el.get_text(strip=True) if el else None

def detail_at(soup, index):
    rows = soup.select(".rentInfoDetail")
    return rows[index].get_text(strip=True) if index < len(rows) else None

def scrape_listing(html):
    soup = BeautifulSoup(html, "html.parser")

    address = soup.select_one(".propertyAddress")
    location = ", ".join(
        s.get_text(strip=True) for s in address.select("span")
    ) if address else None

    amenities = [
        s.get_text(strip=True)
        for s in soup.select("#amenitiesSection .specInfo span")
    ]

    return {
        "name": text_of(soup, "#propertyName"),
        "location": location,
        "rent": detail_at(soup, 0),
        "beds": detail_at(soup, 1),
        "baths": detail_at(soup, 2),
        "size": detail_at(soup, 3),
        "amenities": amenities,
    }

The text_of and detail_at helpers do the same useful thing in two shapes: they query an element and return None when it is missing instead of throwing on a call against nothing. The address is built by joining the text of each span inside .propertyAddress, since Apartments.com splits the street, city, and state into separate elements. Amenities come back as a list because a listing can have anywhere from zero to dozens. That structure keeps the extraction resilient when one field is absent on a given listing, which is common since not every property lists a size or a full amenity set.

Step 3: Put it together

Now wire the fetch and the parse into one runnable script. Fetch the rendered HTML, hand it to the parser, and print the structured record.

import json
from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup

api = CrawlingAPI({"token": "YOUR_CRAWLBASE_JS_TOKEN"})

def crawl(page_url):
    options = {"ajax_wait": "true", "page_wait": 5000}
    response = api.get(page_url, options)
    if response["status_code"] == 200:
        return response["body"].decode("utf-8")
    print(f"Request failed: {response['status_code']}")
    return None

def text_of(soup, selector):
    el = soup.select_one(selector)
    return el.get_text(strip=True) if el else None

def detail_at(soup, index):
    rows = soup.select(".rentInfoDetail")
    return rows[index].get_text(strip=True) if index < len(rows) else None

def scrape_listing(html):
    soup = BeautifulSoup(html, "html.parser")
    address = soup.select_one(".propertyAddress")
    location = ", ".join(
        s.get_text(strip=True) for s in address.select("span")
    ) if address else None
    amenities = [
        s.get_text(strip=True)
        for s in soup.select("#amenitiesSection .specInfo span")
    ]
    return {
        "name": text_of(soup, "#propertyName"),
        "location": location,
        "rent": detail_at(soup, 0),
        "beds": detail_at(soup, 1),
        "baths": detail_at(soup, 2),
        "size": detail_at(soup, 3),
        "amenities": amenities,
    }

def main():
    page_url = "https://www.apartments.com/2630-n-hamlin-ave-chicago-il/kvl7tm9/"
    html = crawl(page_url)
    if not html:
        return
    data = scrape_listing(html)
    print(json.dumps(data, indent=2))

if __name__ == "__main__":
    main()

What the output looks like

Run the full script with python scraper.py and you get a clean structured record for the listing, ready to write to JSON, CSV, or a database.

{
  "name": "2630 N Hamlin Ave",
  "location": "2630 N Hamlin Ave, Chicago, IL, 60647",
  "rent": "$2,350",
  "beds": "2 bd",
  "baths": "1 ba",
  "size": "1,000 sq ft",
  "amenities": ["Air Conditioning", "Dishwasher", "Basement", "Laundry Facilities"]
}

Scaling across listings and pagination

One listing is a demo; a real job runs over a whole search. Apartments.com paginates its search results, so the pattern is two layers: crawl each search-results page to collect the listing URLs, then fetch each listing through the same function you already wrote. Because every listing shares the same structure, the parser works across all of them without changes.

import time

def collect_listing_urls(search_html):
    soup = BeautifulSoup(search_html, "html.parser")
    cards = soup.select("article.placard a.property-link")
    return [a["href"] for a in cards if a.get("href")]

def scrape_search(base_url, pages):
    listings = []
    for page in range(1, pages + 1):
        search_html = crawl(f"{base_url}{page}/")
        if not search_html:
            continue
        for url in collect_listing_urls(search_html):
            html = crawl(url)
            if html:
                listings.append(scrape_listing(html))
            time.sleep(2)
    return listings

results = scrape_search("https://www.apartments.com/chicago-il/", pages=3)
with open("listings.json", "w") as f:
    json.dump(results, f, indent=2)

Apartments.com appends the page number to the search path, so iterating page walks the result set. The time.sleep(2) between listing fetches is deliberate: it paces the run so you are not hammering the site, which is the single most effective habit for staying unblocked. Adjust the page count and city slug to fit your target.

Staying unblocked

Even with rendering handled, Apartments.com watches for scraper-shaped traffic. A few habits keep a run healthy, and they apply to any hard commercial target.

• Pace your requests. Hammering listings in a tight loop is the fastest way to get throttled or served a captcha. Spread requests out, as the sleep above does, and vary your targets instead of crawling one path at full speed.
• Lean on rotation. A pool of residential IPs spreads requests across many real-user addresses so no single one trips a rate limit. The Crawling API handles this for you; if you roll your own stack, this is the part to get right.
• Read the status codes. A run that starts returning challenges or errors is telling you the current rate or IP tier is no longer enough. Treat that as signal to back off, not noise to ignore.

For the broader playbook, see how to scrape websites without getting blocked and the deeper dive on how to bypass captchas while web scraping. If you would rather route your own traffic through a rotating pool instead of using the managed API, the Smart AI Proxy (also called the AI Proxy) gives you the same residential IP rotation as a drop-in proxy endpoint.

Is it legal to scrape Apartments.com?

Whether scraping Apartments.com is allowed depends on Apartments.com's terms of service, your jurisdiction, and what you do with the data. The terms restrict automated access, so scraping can run against those terms regardless of how careful your tooling is. None of the code here changes that; it just makes the technical part work. Read the Apartments.com Terms of Service and its robots.txt, and treat both as the boundary for what you collect.

A few lines worth holding to. Collect only public listing data: the property name and address, rent, beds, baths, square footage, and the publicly shown amenity list that anyone can see without an account. Respect Apartments.com's stated rate expectations and keep your request volume low enough that you are not straining its servers. Avoid anything tied to identifiable individuals, including the contact details of landlords, agents, or property managers listed on a page. If you plan to reuse the data commercially or in bulk, get permission or an official agreement rather than assuming silence is consent.

This guide is deliberately scoped to public listing pages because that is the line that keeps the work defensible. It does not cover anything behind a login, saved-search or account data, the personal or contact details of individuals, login-walled pages, or any attempt to bypass authentication. Public listing data only. If your project needs more than that, a licensing arrangement or a real-estate data provider is the correct path, not a cleverer scraper.

Frequently Asked Questions (FAQs)

Why does a plain fetch return no data from Apartments.com?

Because Apartments.com renders its listing content client-side with JavaScript. The initial HTML is a shell that only fills in after the page's scripts run in a browser, so a raw HTTP request returns status 200 with the rent, beds, baths, and amenity fields blank. To get real data you have to render the page first, which is what the Crawling API's JS token handles for you.

Do I need the normal token or the JS token for Apartments.com?

The JS token. The normal token fetches static HTML, which on Apartments.com is the same empty shell a plain fetch returns. The JS token renders the page in a real browser before handing back the HTML, so the listing fields are present when BeautifulSoup parses them.

What data can I scrape from an Apartments.com listing?

Public listing fields: the property name and street address, the monthly rent or rent range, the number of beds and baths, the square footage, and the amenity list. Stay on data that is visible to any visitor without an account, and avoid the personal contact details of landlords, agents, or property managers, which fall outside the public-listing scope this guide covers.

My selectors return None or an empty list. What changed?

Almost certainly Apartments.com's markup. Its rentInfoDetail rows, the #amenitiesSection wrapper, and the address spans change without notice, so selectors that worked last month can break. Re-inspect a live listing in your browser's dev tools and update the selectors. Periodic selector maintenance is normal for any production scraper.

How do I handle pagination across a city's listings?

Apartments.com appends the page number to the search path, so you crawl each results page in turn, collect the listing links from the cards on it, and fetch each listing with the same parser. Keep a short sleep between requests and stop when a page returns no new cards. The scrape_search function above shows the full loop.

How do I avoid getting blocked while scraping Apartments.com?

Keep your per-IP request rate low, pace requests with a short delay, vary your targets instead of looping one path, and route through rotating residential IPs so no single address trips a rate limit. The Crawling API manages rotation and a trusted IP pool for you; if you build your own stack, that is the part to invest in. Watch the status codes and back off when you start seeing challenges.

HI

Hamza Ikhlaq

Software Developer · Crawlbase

Software developer at Crawlbase writing hands-on guides on scraping target sites, proxies, and the Crawling API.