The Apple App Store is a digital hub where users browse, download, and set up apps on their Apple devices, including iPhones and iPads. With millions of apps, spanning everything from mind-numbing games to productivity apps, and other entertainment stuff that keep us glued to our screens.

If you’re building apps yourself, trying to market something, or just researching market trends, App Store intel can be very useful. The real trick is setting up your scraping approach correctly so you can transform all the data into something that actually helps you make smarter decisions.

So in this blog, we’ll show you how to crawl and scrape Apple App Store data using Crawlbase’s Crawling API and JavaScript. This combination works surprisingly well for gathering details like where apps rank, what their descriptions promise, and what real users are actually saying in reviews.

How to Scrape Apple App Store Data?

Our first step is to create an account with Crawlbase, which will enable us to utilize the Crawling API and serve as our platform for reliably fetching data from the App Store.

Creating a Crawlbase account

  1. Sign up for a Crawlbase account and log in.
  2. Once registered, you’ll receive 1,000 free requests. Add your billing details before using any of the free credits to get an extra 9,000 requests.
  3. Go to your Account Docs and save your Normal Request token for this blog’s purpose.

Setting up the Environment

Next, ensure that Node.js is installed on your device, as it is the backbone of our scraping script, providing a fast JavaScript runtime and access to essential libraries.

Installing Node on Windows:

  1. Go to the official Node.js website and download the Long-Term Support (LTS) version for Windows.
  2. Launch the installer and follow the prompts. Leave the default options selected.
  3. Verify installation by opening a new Command Prompt and running the following commands:
1
2
node -v
npm -v

For macOS:

  1. Go to [https://nodejs.org](https://nodejs.org/) and download the macOS installer (LTS).
  2. Follow the installation wizard.
  3. Open the Terminal and confirm the installation:
1
2
node -v
npm -v

For Linux (Ubuntu/Debian):

  1. Open your terminal to add the NodeSource repository and install Node.js:
1
2
curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
sudo apt-get install -y nodejs
  1. Verify your installation:
1
2
node -v
npm -v

Fetch Script

Grab the script below and save it with a .js extension, any IDE or a coding environment you like will work. Once you’ve saved it, double-check that all the necessary dependencies are installed in your Node.js setup. After that, you should be all set.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import { CrawlingAPI } from 'crawlbase';

const CRAWLBASE_NORMAL_TOKEN = '<Normal requests token>';
const URL = 'https://apps.apple.com/us/app/google-authenticator/id388497605';

async function crawlAppStore() {
  const api = new CrawlingAPI({ token: CRAWLBASE_NORMAL_TOKEN });
  const options = {
    userAgent: 'Mozilla/5.0 (Windows NT 6.2; rv:20.0) Gecko/20121202 Firefox/30.0',
  };

  const response = await api.get(URL, options);

  if (response.statusCode !== 200) {
    throw new Error(`Request failed with status code: ${response.statusCode}`);
  }

  return response.body;
}

IMPORTANT: Make sure to replace <Normal requests token> with your actual Crawlbase normal request token before running the script.

This script shows how to use Crawlbase’s Crawling API to retrieve HTML content from the Apple App Store without getting blocked. Note that the response hasn’t been scraped yet. We still need to remove unnecessary elements, clean the data, and produce a parsed, structured response.

Locating specific CSS selectors

Now that you understand how to send a simple API request using Node.js, let’s locate the data we need from our target URL so we can later write code to clean and parse it.

The first thing you’ll notice is the main section at the top. It’s usually where we’ll find the most important details and is typically well-structured, making it an ideal target for scraping.

Go ahead and open your target URL and locate each selector. For example, let’s search for the title:

An image of the Chrome Browser Developer Tools with the content header element highlighted.

Take note of the .app-header__title and do the same for subtitle, seller, category, stars, rating, and price. Once that’s done, this section is complete.

The process is pretty much the same for the rest of the page. Here’s another example: if you want to include the customer average rating in the Ratings and Reviews section, right-click on the data and select Inspect:

An image of the Chrome Browser Developer Tools highlighting the content ratings element.

You know the gist. It should now be a piece of cake for you to locate the remaining data you need.

Parsing the HTML in Node.js

Now that you’re an expert in extracting the CSS selectors, it is time to build the code to parse the HTML. This is where Cheerio comes in. It is a lightweight and powerful library that enables us to select relevant data from the HTML source code within Node.js.

Start by creating your project folder and run:

1
2
npm init -y
npm install crawlbase lodash casenator cheerio

Import the Required Libraries

Then in your .js file, import the required libraries for this project, including Cheerio:

1
2
3
4
import _ from 'lodash';
import { CrawlingAPI } from 'crawlbase';
import { toCamelCase } from 'casenator';
import * as cheerio from 'cheerio';

Don’t forget to set up the Crawling API as well as the target website:

1
2
const CRAWLBASE_NORMAL_TOKEN = '<Normal requests token>';
const URL = 'https://apps.apple.com/us/app/google-authenticator/id388497605';

Functions for Scraping Apple Store Data

This is where we’ll use the CSS selectors we’ve collected earlier. Let’s write the part of the code that pulls the bits of information from the App Store page.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
function scrapePrimaryAppDetails($) {
  let title = $('.app-header__title').text().trim();
  const titleBadge = $('.badge--product-title').text().trim();
  title = title.replace(titleBadge, '').trim();
  const subtitle = $('.app-header__subtitle').text().trim();
  const seller = $('.app-header__identity').text().trim();
  let category = null;
  try {
    category = $('.product-header__list__item a.inline-list__item').text().trim().split('in')[1].trim();
  } catch {
    category = null;
  }
  const stars = $('.we-star-rating').attr('aria-label');
  const rating = $('.we-rating-count').text().trim().split('•')[1].trim();
  const price = $('.app-header__list__item--price').text().trim();

  return { title, subtitle, seller, category, stars, rating, price };
}

Just like that, it will extract the title, subtitle, seller, category, star rating, overall ratings, and price.

From this point, you can add more functions for each section of the page. You can add the Preview Image and Description, as well as user reviews, etc.

Combine Everything in One Function

Once the scraper is complete, we need to combine everything in one function and print the result:

1
2
3
4
5
6
7
8
9
10
function scrapeAppStore(html) {
  const $ = cheerio.load(html);
  return {
    primaryAppDetails: scrapePrimaryAppDetails($),
    appPreviewAndDescription: scrapeAppPreviewAndDescription($),
    ratingsAndReviews: { reviews: scrapeRatingsAndReviews($) },
    informationSection: scrapeInformationSection($),
    relatedAppsAndRecommendations: scrapeRelatedAppsAndRecommendations($),
  };
}

Complete Code to Scrape Apple App Store Data

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
import _ from 'lodash';
import { CrawlingAPI } from 'crawlbase';
import { toCamelCase } from 'casenator';
import * as cheerio from 'cheerio';

const CRAWLBASE_NORMAL_TOKEN = '<Normal requests token>';
const URL = 'https://apps.apple.com/us/app/google-authenticator/id388497605';

async function crawlAppStore() {
  const api = new CrawlingAPI({ token: CRAWLBASE_NORMAL_TOKEN });
  const options = {
    userAgent: 'Mozilla/5.0 (Windows NT 6.2; rv:20.0) Gecko/20121202 Firefox/30.0',
  };

  const response = await api.get(URL, options);

  if (response.statusCode !== 200) {
    throw new Error(`Request failed with status code: ${response.statusCode}`);
  }
  return response.body;
}

function scrapePrimaryAppDetails($) {
  let title = $('.app-header__title').text().trim();
  const titleBadge = $('.badge--product-title').text().trim();
  title = title.replace(titleBadge, '').trim();
  const subtitle = $('.app-header__subtitle').text().trim();
  const seller = $('.app-header__identity').text().trim();
  let category = null;
  try {
    category = $('.product-header__list__item a.inline-list__item').text().trim().split('in')[1].trim();
  } catch {
    category = null;
  }
  const stars = $('.we-star-rating').attr('aria-label');
  const rating = $('.we-rating-count').text().trim().split('•')[1].trim();
  const price = $('.app-header__list__item--price').text().trim();

  return { title, subtitle, seller, category, stars, rating, price };
}

function scrapeAppPreviewAndDescription($) {
  const sources = $('source').toArray();
  const imageUrl =
    sources
      .map((element) => $(element).attr('srcset'))
      .filter((srcset) => srcset)
      .map((srcset) => srcset.split(',')[0].trim().split(' ')[0])
      .find((url) => url) || null;
  let appDescription = $('.section__description').text().trim();
  appDescription = appDescription.replace(/^Description\s*/, '');

  return { imageUrl, appDescription };
}

function scrapeRatingsAndReviews($) {
  const reviews = [];
  $('.we-customer-review').each((index, element) => {
    const stars = $(element).find('.we-star-rating').attr('aria-label');
    const reviewerName = $(element).find('.we-customer-review__user').text().trim();
    const reviewTitle = $(element).find('.we-customer-review__title').text().trim();
    const fullReviewText = $(element).find('.we-customer-review__body').text().trim();
    const reviewDate = $(element).find('.we-customer-review__date').attr('datetime');
    reviews.push({ stars, reviewerName, reviewTitle, fullReviewText, reviewDate });
  });

  return reviews;
}

function scrapeInformationSection($) {
  const information = {};
  $('dl.information-list dt').each((index, element) => {
    const key = $(element).text().trim();
    const value = $(element).next('dd').text().trim();
    if (key && value) {
      const camelKey = toCamelCase(key);
      if (camelKey === 'languages') {
        information[camelKey] = _.uniq(value.split(',').map((item) => item.trim())).sort();
      } else if (camelKey === 'compatibility') {
        information[camelKey] = _.uniq(
          value
            .split('\n')
            .map((item) => item.trim())
            .filter((item) => item),
        ).sort();
      } else {
        information[camelKey] = value;
      }
    }
  });

  return information;
}

function scrapeRelatedAppsAndRecommendations($) {
  function extractAppsFromSection(headlineText) {
    const results = [];
    $('h2.section__headline').each((index, element) => {
      const currentHeadlineText = $(element).text().trim();
      if (currentHeadlineText === headlineText) {
        const parent = $(element).parent();
        const nextSibling = parent.next();
        nextSibling.find('a.we-lockup--in-app-shelf').each((appIndex, appElement) => {
          const appTitle = $(appElement).find('.we-lockup__title').text().trim();
          const appUrl = $(appElement).attr('href');
          if (appTitle && appUrl) {
            results.push({
              title: appTitle,
              url: appUrl,
            });
          }
        });
      }
    });

    return results;
  }

  return {
    developerApps: extractAppsFromSection('More By This Developer'),
    relatedApps: extractAppsFromSection('You Might Also Like'),
  };
}

function scrapeAppStore(html) {
  const $ = cheerio.load(html);
  const data = {
    primaryAppDetails: {
      ...scrapePrimaryAppDetails($),
    },
    appPreviewAndDescription: {
      ...scrapeAppPreviewAndDescription($),
    },
    ratingsAndReviews: {
      reviews: scrapeRatingsAndReviews($),
    },
    informationSection: {
      ...scrapeInformationSection($),
    },
    relatedAppsAndRecommendations: {
      ...scrapeRelatedAppsAndRecommendations($),
    },
  };

  return data;
}

const html = await crawlAppStore();
const data = scrapeAppStore(html);
console.log(JSON.stringify(data, null, 2));

And when you run your script:

1
npm run crawl

You’ll see the output in this structure:

An image displaying a structured JSON output obtained from scraping the Apple App Store.

This organized structure provides a solid foundation for further analysis, reporting, or visualization, regardless of your end goal.

Check out the complete code in our GitHub repository for this blog.

Scrape Apple Store Data with Crawlbase

Scraping the Apple App Store can provide valuable insights into how apps are presented, user responses, and the performance of competitors. With Crawlbase and a solid HTML parser like Cheerio, you can automate the extraction of Apple data and turn it into something actionable.

For tracking reviews, comparing prices, or just exploring the app ecosystem, this setup can save you time and effort while delivering the data you need.

Start your next scraping project now with Crawlbase’s Smart Proxy and Crawling API to avoid getting blocked!

Frequently Asked Questions

Q: Can I scrape any app on the App Store?

A. Yes, as long as you have the app’s public URL. Apple doesn’t provide a complete public index, so you’ll need to build your list or collect links from other sources.

A. It’s usually acceptable to scrape public data for research or personal use, but make sure your usage complies with Apple’s Terms of Service. Steer clear of excessive scraping and usage restrictions.

Q. What if I get blocked or rate-limited?

A. If too many requests are sent from the same IP address or if the behavior seems automated, scraping websites may be blocked or have their rates limited. To avoid such issues, you can utilize Crawlbase’s Crawling API and Smart Proxy. They include anti-block features like IP geo-location and rotation, which significantly reduce the chances of being blocked and enable more accurate data collection.