How to Scrape the Apple App Storeratings, reviews, and metadata

The Apple App Store holds millions of app listings, and each public listing carries the kind of structured detail product teams, marketers, and analysts care about: the app name, its developer, the star rating, how many reviews it has, the category it sits in, and the price. Pulling that data by hand does not scale past a handful of apps, so the practical move is to build an App Store scraper in Python that fetches a listing page, parses the fields you want, and hands you a clean record.

This guide walks through exactly that. You will fetch rendered App Store listing pages through the Crawling API using the official crawlbase Python client, parse the markup with BeautifulSoup, and extract a structured row for each app. We keep the whole walkthrough scoped to public listing data, and the legality section near the end is not boilerplate, so read it before you point this at real volume.

What you will build

A Python script that takes a public App Store app URL, retrieves the rendered HTML through the Crawling API, and extracts a structured record. We will use a well-known free app as the running example and pull these fields:

• App name the listing title, for example "Microsoft Authenticator".
• Developer the publisher or seller, like "Microsoft Corporation".
• Rating the average star score, such as "4.8".
• Review count the number of ratings behind that score.
• Category the App Store category the app is filed under.
• Price the listed price, or "Free" when there is none.

Why a plain fetch struggles on the App Store

If you request an App Store listing URL with a bare HTTP client, you run into two problems. First, the listing page leans on JavaScript to fill in parts of its content, so the raw HTML you get back can be missing the fields you came for. Second, Apple watches for automated traffic: datacenter IPs and request patterns that do not look like a real browser get challenged or throttled before they reach the useful markup.

So a working App Store scraper needs two things in one request: a browser that actually renders the page, and an IP the platform reads as a real visitor. You can assemble that yourself with a headless browser plus a pool of rotating residential proxies, but stitching those together and keeping them healthy is most of the work. The Crawling API folds both into a single call: you send it the URL, it renders the page behind a trusted IP, and it returns finished HTML for you to parse.

Prerequisites

You need a few things in place before writing any code. None of them take long.

Basic Python. You should be comfortable writing and running a Python script and installing packages with pip. If you are new to the language, the official Python docs and any beginner course will get you to the level this tutorial assumes.

Python 3.8 or later. Confirm your version with python --version. If you do not have it, install it from python.org or through a distribution like Anaconda.

A Crawlbase account and token. Sign up, open your dashboard, and copy your token from the account docs page. Use the JavaScript (JS) token for App Store listings. Treat the token like a password: it authenticates your requests, so keep it out of version control.

Set up the project

Create a virtual environment so project dependencies stay isolated, then install the two libraries the scraper needs.

python --version

python -m venv appstore_env
source appstore_env/bin/activate

pip install crawlbase beautifulsoup4

On Windows, activate the environment with appstore_env\Scripts\activate instead of the source line. Two dependencies do the work: crawlbase is the official client for the Crawling API, and beautifulsoup4 parses the returned HTML so you can pull out individual fields by CSS selector. If you want a refresher on the parsing library, the guide to BeautifulSoup in Python covers the selectors used below.

Step 1: Fetch the rendered listing

Start by getting the finished page. Import the CrawlingAPI class, initialize it with your token, and request the app URL. Checking the status code before you parse keeps failures loud instead of silent.

from crawlbase import CrawlingAPI

api = CrawlingAPI({"token": "YOUR_CRAWLBASE_JS_TOKEN"})

def crawl(page_url):
    options = {"ajax_wait": "true", "page_wait": 3000}
    response = api.get(page_url, options)
    if response["status_code"] == 200:
        return response["body"].decode("utf-8")
    print(f"Request failed: {response['status_code']}")
    return None

if __name__ == "__main__":
    page_url = "https://apps.apple.com/us/app/microsoft-authenticator/id983156458"
    html = crawl(page_url)
    print(html[:500] if html else "No HTML returned")

The two wait options help with elements that render late. ajax_wait tells the API to wait for asynchronous content to finish loading, and page_wait holds for a fixed number of milliseconds after load so late-rendering parts appear before the page is captured. Three seconds is a reasonable start; raise it if fields come back empty. Run the script with python scraper.py and you should see real listing markup, not a stripped-down shell. That confirms rendering works before you write a single selector.

Step 2: Parse the app fields with BeautifulSoup

With rendered HTML in hand, load it into BeautifulSoup and pull each field by its selector. App Store listings lay the core details out in a predictable structure, so you can map name, developer, rating, review count, category, and price to individual selectors. Wrap the extraction in helpers so one missing field does not crash the run.

from bs4 import BeautifulSoup

def text_of(soup, selector):
    el = soup.select_one(selector)
    return el.get_text(strip=True) if el else None

def scrape_app(html):
    soup = BeautifulSoup(html, "html.parser")

    rating_text = text_of(soup, ".we-rating-count.star-rating__count")
    stars, reviews = (rating_text.split(" • ") + [None, None])[:2] if rating_text else (None, None)

    return {
        "name": text_of(soup, ".product-header__title"),
        "developer": text_of(soup, ".product-header__identity a"),
        "rating": stars,
        "review_count": reviews,
        "category": text_of(soup, ".information-list__item__term:-soup-contains('Category') + dd a"),
        "price": text_of(soup, ".app-header__list__item--price"),
    }

The text_of helper does two useful things at once: it queries a single element and returns None when the element is missing, instead of throwing on a .get_text() call against nothing. That keeps the extraction resilient when one field is absent on a given listing. The rating string on the App Store packs both the star score and the review count into one value separated by a bullet, so we split it once and assign both parts, defaulting to None when the format does not match.

Step 3: Put it together

Now wire the fetch and the parse into one runnable script. Fetch the rendered HTML, hand it to the parser, and print the structured record.

import json
from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup

api = CrawlingAPI({"token": "YOUR_CRAWLBASE_JS_TOKEN"})

def crawl(page_url):
    options = {"ajax_wait": "true", "page_wait": 3000}
    response = api.get(page_url, options)
    if response["status_code"] == 200:
        return response["body"].decode("utf-8")
    print(f"Request failed: {response['status_code']}")
    return None

def text_of(soup, selector):
    el = soup.select_one(selector)
    return el.get_text(strip=True) if el else None

def scrape_app(html):
    soup = BeautifulSoup(html, "html.parser")
    rating_text = text_of(soup, ".we-rating-count.star-rating__count")
    stars, reviews = (rating_text.split(" • ") + [None, None])[:2] if rating_text else (None, None)

    return {
        "name": text_of(soup, ".product-header__title"),
        "developer": text_of(soup, ".product-header__identity a"),
        "rating": stars,
        "review_count": reviews,
        "category": text_of(soup, ".information-list__item__term:-soup-contains('Category') + dd a"),
        "price": text_of(soup, ".app-header__list__item--price"),
    }

def main():
    page_url = "https://apps.apple.com/us/app/microsoft-authenticator/id983156458"
    html = crawl(page_url)
    if not html:
        return
    data = scrape_app(html)
    print(json.dumps(data, indent=2))

if __name__ == "__main__":
    main()

What the output looks like

Run the full script with python scraper.py and you get a clean structured record for the app, ready to write to JSON, CSV, or a database.

{
  "name": "Microsoft Authenticator",
  "developer": "Microsoft Corporation",
  "rating": "4.8",
  "review_count": "339.5K Ratings",
  "category": "Productivity",
  "price": "Free"
}

Scaling to many apps

One listing is a demo; a real job runs over a list of apps. The shape stays the same: keep a list of app URLs, fetch each through the Crawling API, parse it with the same function, and collect the rows. Because every listing shares the same structure, the parser you already wrote works across all of them without changes. The detail that matters at volume is pacing: space the requests out so you read like a stream of normal visitors rather than a tight loop.

import time

apps = [
    "https://apps.apple.com/us/app/microsoft-authenticator/id983156458",
    "https://apps.apple.com/us/app/google-authenticator/id388497605",
]

results = []
for url in apps:
    html = crawl(url)
    if html:
        results.append(scrape_app(html))
    time.sleep(2)

with open("apps.json", "w") as f:
    json.dump(results, f, indent=2)

The time.sleep call between requests is deliberate. Even though the Crawling API rotates IPs for you, pacing keeps your run polite and predictable, and it gives each rendered page time to come back without piling requests on top of one another. If you want to capture per-app reviews as well, the same fetch-then-parse pattern extends to the review section; the guide to scraping customer reviews covers that shape in more depth.

Structured app data without scraping

Scraping the public listing page is the right tool when you need exactly what a visitor sees and there is no API that returns it. But for plain structured app metadata, Apple offers official channels that are cleaner and more stable than parsing markup, and you should prefer them when they cover your need.

• iTunes Search API and App Store RSS feeds. Apple's public iTunes Search API returns app details (name, developer, average rating, rating count, category, price, and more) as JSON, and the App Store RSS feeds publish ranked lists of top free, paid, and trending apps. For structured metadata at the listing level, these are the first thing to reach for.
• App Store Connect API. If the apps are your own, the App Store Connect API gives you first-party access to your listings, sales, and analytics under your developer account, with no scraping involved.

Reach for the scraper when you need data the official feeds do not expose, or when you are assembling a view across many third-party listings. For everything the iTunes Search API and RSS feeds already return in JSON, use them first.

Staying unblocked

Even with rendering handled, Apple watches for scraper-shaped traffic. A few habits keep a run healthy, and they apply to any hard commercial target.

• Pace your requests. Hammering listings in a tight loop is the fastest way to get throttled. Spread requests out and vary your targets instead of crawling one path at full speed.
• Lean on rotation. A pool of residential IPs spreads requests across many real-user addresses so no single one trips a rate limit. The Crawling API handles this for you; if you roll your own stack, this is the part to get right.
• Read the status codes. A run that starts returning challenges or errors is telling you the current rate or IP tier is no longer enough. Treat that as signal to back off, not noise to ignore.

For the broader playbook, see the deep dive on how to bypass captchas while web scraping and the guide to scraping JavaScript pages with Python. If you would rather route your own traffic through a rotating pool instead of using the managed API, the Smart AI Proxy (also called the AI Proxy) gives you the same residential IP rotation as a drop-in proxy endpoint.

Is it legal to scrape the App Store?

Whether scraping the App Store is allowed depends on Apple's terms of service, your jurisdiction, and what you do with the data. Apple's terms restrict automated access, so scraping can run against those terms regardless of how careful your tooling is. None of the code here changes that; it just makes the technical part work. Read Apple's Terms of Service and the App Store robots.txt, and treat both as the boundary for what you collect.

A few lines worth holding to. Collect only public app-listing data: the app name, developer, rating, review count, category, and price that anyone can see without an account. Respect Apple's stated rate expectations and keep your request volume low enough that you are not straining its servers. This guide is deliberately scoped to public listing pages because that is the line that keeps the work defensible. It does not cover account or personal data, login-walled pages, anything behind a sign-in, or any attempt to bypass authentication. Those are out of scope here, and reaching them runs against Apple's terms.

For structured app data, prefer Apple's official channels. The App Store Connect API gives first-party access to your own apps, and the public iTunes Search API and App Store RSS feeds return app metadata as JSON without scraping a single page. When an official feed covers your need, it is the right tool; the scraper in this guide is for public listing data the feeds do not expose, nothing more.

Frequently Asked Questions (FAQs)

Why does a plain fetch return incomplete data from the App Store?

Because App Store listings render parts of their content client-side with JavaScript. A raw HTTP request can come back with the page shell but missing fields like rating or category, since those fill in only after the page's scripts run in a browser. The Crawling API's JS token renders the page first, so the fields are present when BeautifulSoup parses them.

Do I need the normal token or the JS token for the App Store?

Use the JS token. The normal token fetches static HTML, which on the App Store can leave some listing fields empty. The JS token renders the page in a real browser before handing back the HTML, so the name, developer, rating, review count, category, and price are present when you parse.

Does Apple have an official API for app data?

Yes. The public iTunes Search API returns app details as JSON (name, developer, average rating, rating count, category, price, and more), and the App Store RSS feeds publish ranked top-app lists. For your own apps, the App Store Connect API gives first-party access to listings and analytics. Prefer these official channels for structured metadata, and reach for scraping only when you need public listing data the feeds do not expose.

My selectors return None. What changed?

Almost certainly Apple's markup. The we-rating-count markers, product-header elements, and information-list rows change without notice, so selectors that worked last month can break. Re-inspect a live listing in your browser's dev tools and update the selectors. Periodic selector maintenance is normal for any production scraper.

Can I scrape account or personal data from the App Store?

No, and this guide does not cover it. Account data, purchase history, and anything behind a sign-in are not public data. Scraping login-walled content, or bypassing authentication to reach it, is out of scope here and runs against Apple's terms. For first-party data about your own apps, the App Store Connect API is the correct route.

How do I avoid getting blocked while scraping the App Store?

Keep your per-IP request rate low, vary your targets instead of looping one path, and route through rotating residential IPs so no single address trips a rate limit. The Crawling API manages rotation and a trusted IP pool for you; if you build your own stack, that is the part to invest in. Watch the status codes and back off when you start seeing challenges.

HI

Hamza Ikhlaq

Software Developer · Crawlbase

Software developer at Crawlbase writing hands-on guides on scraping target sites, proxies, and the Crawling API.