Google Shopping stands out as one of the most data-rich e-commerce platforms. Its vast collection of products, prices, and retailers makes it a goldmine for companies and data enthusiasts alike.

Google Shopping plays a crucial role for online buyers and sellers. By 2024, it will offer millions of items from numerous retailers across the globe, giving shoppers a wide range of choices and bargains. When you extract data from Google Shopping, you gain insights into product costs, stock levels, and rival offerings, which helps you to make choices based on facts.

This post will show you how to scrape Google Shopping data with Python. We’ll use the Crawlbase Crawling API to get around restrictions and collect the information.

Here’s a breakdown of our step-by-step plan.

Table Of Content

  1. Why Scrape Google Shopping?
  • Benefits of scraping Google Shopping
  • Key Data Points of Google Shopping
  1. Bypass Limitation with Crawlbase Crawling API
  2. Prerequisites
  • Setting Up Your Python Environment
  • Installing Required Libraries
  1. Google Shopping SERP Structure
  • Key Elements of Google Shopping SERP
  1. Scraping Google Shopping SERP
  • Inspecting the HTML Structure
  • Writing Google Shopping SERP Scraper
  • Handling Pagination
  • Saving Data to JSON File
  • Complete Code
  1. Google Shopping Product Page Structure
  • Key Elements of Google Shopping Product Page
  1. Scraping Google Shopping Product Page
  • Inspecting the HTML Structure
  • Writing Google Shopping Product Page Scraper
  • Saving Data to JSON File
  • Complete Code
  1. Final Thoughts
  2. Frequently Asked Questions

Why Scrape Google Shopping?

Scraping Google Shopping lets you get useful insights. These insights help shape your business plan, make your products better, and set the right prices. In this part, we’ll look at the good things about getting data from Google Shopping and the key bits of info you can pull out.

Benefits of Scraping Google Shopping

Benefits of Scraping Google Shipping

Competitive Pricing Analysis

Pricing is one of key factors for customers’ decision. By scraping Google Shopping, you can see your competitor’s prices in real-time and adjust your pricing accordingly. This ensures your prices are competitive and attract more customers and sales.

Product Availability Monitoring

Product availability is key to inventory management and meeting customer demand. Scraping Google Shopping lets you see which products are in stock, out of stock or on sale. This will help you optimize your inventory so you have the right products at the right time.

Trend Analysis and Market Insights

Staying on top of the trends is vital for any e-commerce business. By scraping Google Shopping, you can see emerging trends, popular products, and shifting customer behavior. This will help your product development, marketing strategies and business decisions.

Improving Product Listings

Detailed and beautiful product listings are the key to converting browsers into buyers. By looking at successful listings on Google Shopping, you can get ideas for your product descriptions, images, and keywords. This will help your rankings and visibility.

Key Data Points of Google Shopping

When scraping Google Shopping, you can extract the following data points:

  • Product Titles and Descriptions: See how competitors are presenting their products and refine your product listings to get more customers.
  • Prices and Discounts: Extract helpful information on prices, including discounts and special offers, to monitor competitors’ pricing strategies. You can use this data to adjust your prices to stay competitive and sell more.
  • Product Ratings and Reviews: Customer ratings and reviews give insight into customer satisfaction and product quality. You can analyze their feedback to see the strengths and weaknesses of your products.
  • Retailer Information: Extract information about retailers selling similar products to see who the key players are in your market and potential partners.
  • Product Categories and Tags: See how products are categorized and tagged to improve your product organization and search engine optimization (SEO) so customers can find your products.
  • Images and Visual Content: Images are crucial for capturing customer interest. By examining visual content from top-performing listings, you can enhance the quality of your product images to improve engagement.

Collecting and analyzing these data points enables you to make informed decisions that drive your business forward. In the next section, we’ll discuss how to overcome challenges in web scraping using the Crawlbase Crawling API.

Bypass Limitations with Crawlbase Crawling API

Web scraping is a powerful tool for gathering data, but it comes with challenges like IP blocking, rate limits, dynamic content, and regional differences. The Crawlbase Crawling API helps overcome these issues, making the scraping process smoother and more effective.

Bypass Limitations with Crawlbase Crawling API

IP Blocking and Rate Limiting

Websites may block IP addresses that send too many requests quickly, a problem known as rate limiting. Crawlbase Crawling API helps by rotating IP addresses and controlling request speeds, enabling you to scrape data without interruptions.

Dynamic Content and JavaScript

Many websites use JavaScript to load content after the page has initially loaded. Traditional scraping methods might miss this dynamic data. The Crawlbase Crawling API can handle JavaScript, ensuring you get all the content on the page, even the elements that appear later.

CAPTCHA and Anti-Bot Measures

To prevent automated scraping, websites often use captchas and other anti-bot measures. The Crawlbase Crawling API can get past these barriers, allowing you to keep collecting data without running into these obstacles.

Geolocation and Country-Specific Data

Websites sometimes show different content based on the user’s location. The Crawlbase Crawling API lets you choose the country for your requests so you can get data specific to a particular region, which is helpful for localized product information and pricing.

Crawlbase Crawling API handles these shared web scraping challenges effectively and collects valuable data from Google Shopping without issues. In the next section, we’ll discuss what you need to set up your Python environment for scraping.

Prerequisites

Before you begin scraping Google Shopping data, you need to set up your Python environment and install the necessary libraries. This section will walk you through the essential steps to get everything ready for your web scraping project.

Setting Up Your Python Environment

Install Python

Ensure that Python is installed on your computer. Python is a popular programming language used for web scraping and data analysis. If you don’t have Python installed, download it from the official Python website. Follow the installation instructions for your operating system.

Create a Virtual Environment

Creating a virtual environment is a good practice to keep your project dependencies organized and avoid conflicts with other projects. To create a virtual environment, open your command line or terminal and run:

1
python -m venv myenv

Replace myenv with a name for your environment. To activate the virtual environment, use the following command:

  • On Windows:

    1
    myenv\Scripts\activate

  • On macOS and Linux:

    1
    source myenv/bin/activate

Installing Required Libraries

With your virtual environment set up, you need to install the following libraries for web scraping and data processing:

BeautifulSoup4

The BeautifulSoup4 library helps with parsing HTML and extracting data. It works well with the Crawlbase library for efficient data extraction. Install it by running:

1
pip install beautifulsoup4

Crawlbase

The Crawlbase library allows you to interact with the Crawlbase products. It helps handle challenges like IP blocking and dynamic content. Install it by running:

1
pip install crawlbase

Note: To access Crawlbase Crawling API, you need a token. You can get one by creating an account on Crawlbase. Crawlbase provides two types of tokens: a Normal Token for static websites and a JavaScript (JS) Token for handling dynamic or browser-based requests. In case of Google Shopping, you need Normal Token. The first 1,000 requests are free to get you started, with no credit card required.

With these libraries installed, you’re ready to start scraping Google Shopping data. In the next section, we’ll dive into the structure of the Google Shopping search results page and how to identify the data you need to extract.

Google Shopping SERP Structure

Knowing the structure of the Google Shopping Search Engine Results Page (SERP) is key to web scraping. This helps you find and extract the data you need.

Key Elements of Google Shopping SERP

Google Shopping SERP

1. Product Listings

Each product listing has:

  • Product Title: The name of the product.
  • Product Image: The image of the product.
  • Price: The price of the product.
  • Retailer Name: The store or retailer selling the product.
  • Ratings and Reviews: Customer reviews if available.

2. Pagination

Google Shopping results are often spread across multiple pages. Pagination links allow you to get to more product listings, so you need to scrape data from all pages for full results.

3. Filters and Sorting Options

Users can refine search results by applying filters like price range, brand, or category. These will change the content displayed and are important for targeted data collection.

4. Sponsored Listings

Some products are marked as sponsored or ads and are displayed prominently on the page. If you only want non-sponsored products, you need to be able to tell the difference between sponsored and organic listings.

Next up, we’ll show you how to write a scraper for the Google Shopping SERP and save the data to a JSON.

Scraping Google Shopping SERP

In this section, we’ll go through how to scrape the Google Shopping Search Engine Results Page (SERP) for product data. We’ll cover inspecting the HTML, writing the scraper, pagination and saving the data to a JSON file.

Inspecting the HTML Structure

Before you write your scraper, use your browser’s Developer Tools to inspect the Google Shopping SERP.

Google Shopping SERP Inspect
  • Right-click on a product listing and select “Inspect” to open the Developer Tools.
  • Hover over elements in the “Elements” tab to see which part of the page they correspond to.
  • Identify the CSS selectors for elements like product title, price, and retailer name.

Writing Google Shopping SERP Scraper

To start scraping, we’ll use the Crawlbase Crawling API to fetch the HTML content. Below is an example of how to set up the scraper for the search query “louis vuitton bags”:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup
import json

# Initialize CrawlingAPI with your access token
crawling_api = CrawlingAPI({'token': 'CRAWLBASE_TOKEN'})

# Set options for the request
options = {
    'country': 'FR',  # Adjust the country code as needed
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
}

# Function to scrape Google Shopping SERP
def scrape_google_shopping(url):
    response = crawling_api.get(url, options)
    if response['headers']['pc_status'] == '200':
        html_content = response['body'].decode('utf-8')
        soup = BeautifulSoup(html_content, 'html.parser')

        products = []
        for item in soup.select('.sh-dgr__grid-result'):
            title = item.select_one('h3.tAxDx').text.strip() if item.select_one('h3.tAxDx') else None
            price = item.select_one('span.a8Pemb.OFFNJ').text.strip() if item.select_one('span.a8Pemb.OFFNJ') else None
            image = item.select_one('div.FM6uVc > div.ArOc1c > img')['src'] if item.select_one('div.FM6uVc > div.ArOc1c > img') else None
            retailer = item.select_one('.aULzUe.IuHnof').text.strip() if item.select_one('.aULzUe.IuHnof') else None
            product_url = 'https://www.google.com' + item.select_one('a.Lq5OHe')['href'] if item.select_one('a.Lq5OHe') else None

            products.append({
                'title': title,
                'price': price,
                'image': image,
                'retailer': retailer,
                'product_url': product_url
            })
        return products
    else:
        print(f"Failed to fetch the page. Status code: {response['headers']['pc_status']}")
        return []

# Example URL for scraping
url = 'https://www.google.com/search?q=louis+vuitton+bags&tbm=shop&start=0&num=20'

Handling Pagination

To scrape multiple pages, you need to modify the start parameter in the URL. This parameter controls the starting index for the results. For instance, to scrape the second page, set start=20, the third page start=40, and so on.

1
2
3
4
5
6
7
8
9
10
11
def scrape_multiple_pages(base_url, pages=3):
    all_products = []
    for page in range(pages):
        start_index = page * 20
        paginated_url = f"{base_url}&start={start_index}"
        products = scrape_google_shopping(paginated_url)
        all_products.extend(products)
    return all_products

# Scrape multiple pages
all_products = scrape_multiple_pages(url, pages=3)

Saving Data to JSON File

After extracting the data, you can save it to a JSON file for further analysis or processing:

1
2
3
4
5
6
7
def save_to_json(data, filename='products.json'):
    with open(filename, 'w', encoding='utf-8') as f:
        json.dump(data, f, ensure_ascii=False, indent=4)

# Save products to JSON
if all_products:
  save_to_json(all_products)

Complete Code

Here’s the complete code to scrape Google Shopping SERP, handle pagination, and save the data to a JSON file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup
import json

# Initialize CrawlingAPI with your access token
crawling_api = CrawlingAPI({'token': 'CRAWLBASE_TOKEN'})

options = {
    'country': 'FR',
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
}

def scrape_google_shopping(url):
    response = crawling_api.get(url, options)
    if response['headers']['pc_status'] == '200':
        html_content = response['body'].decode('utf-8')
        soup = BeautifulSoup(html_content, 'html.parser')

        products = []
        for item in soup.select('.sh-dgr__grid-result'):
            title = item.select_one('h3.tAxDx').text.strip() if item.select_one('h3.tAxDx') else None
            price = item.select_one('span.a8Pemb.OFFNJ').text.strip() if item.select_one('span.a8Pemb.OFFNJ') else None
            image = item.select_one('div.FM6uVc > div.ArOc1c > img')['src'] if item.select_one('div.FM6uVc > div.ArOc1c > img') else None
            retailer = item.select_one('.aULzUe.IuHnof').text.strip() if item.select_one('.aULzUe.IuHnof') else None
            product_url = 'https://www.google.com' + item.select_one('a.Lq5OHe')['href'] if item.select_one('a.Lq5OHe') else None

            products.append({
                'title': title,
                'price': price,
                'image': image,
                'retailer': retailer,
                'product_url': product_url
            })
        return products
    else:
        print(f"Failed to fetch the page. Status code: {response['headers']['pc_status']}")
        return []

def scrape_multiple_pages(base_url, pages=3):
    all_products = []
    for page in range(pages):
        start_index = page * 20
        paginated_url = f"{base_url}&start={start_index}"
        products = scrape_google_shopping(paginated_url)
        all_products.extend(products)
    return all_products

def save_to_json(data, filename='products.json'):
    with open(filename, 'w', encoding='utf-8') as f:
        json.dump(data, f, ensure_ascii=False, indent=4)

url = 'https://www.google.com/search?q=louis+vuitton+bags&tbm=shop&start=0&num=20'
all_products = scrape_multiple_pages(url, pages=3)

if all_products:
  save_to_json(all_products)

Example Output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
[
    {
        "title": "Louis Vuitton Mini Pochette Accessoires Monogram",
        "price": "$760.00",
        "image": "https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcSobTmaRiPTU7ZgclxBAEGJXEqNJmaDrRPTf0yEAm4vAq4z5X5cnK_NeAWP28Hpr7OEfA7_HwkXDUV_JgjgJtJLcqctrK1xBnnp8C9HhTyAK1ByCfQOD9XTuSgq&usqp=CAE",
        "retailer": "louisvuitton.com",
        "product_url": "https://www.google.com/shopping/product/11460745201866483383?q=louis+vuitton+bags&num=20&udm=28&prds=eto:624870836277486981_0,pid:15363890291064430132&sa=X&ved=0ahUKEwjOzJ7ex9uHAxUvL0QIHcJdOsEQ8wIIlgc"
    },
    {
        "title": "Louis Vuitton Graceful mm Peony Monogram",
        "price": "$1,890.00",
        "image": "https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcRB4Js18nK-0mJ1b1e_wxQJL_odHuiA9gxpiBTOPumm0kjw1CKqKtKcIkPWRy3XrPwuRVhFlCEfcXHl4wdXMOz-ibqUICzx0mGgNR-3eKSVg0WXepQxqxWt1dvX&usqp=CAE",
        "retailer": "louisvuitton.com",
        "product_url": "https://www.google.com/shopping/product/14578373001752515304?q=louis+vuitton+bags&num=20&udm=28&prds=eto:9994313334712804353_0,pid:1835053385849536242,rsk:PC_14933825504187696392&sa=X&ved=0ahUKEwjOzJ7ex9uHAxUvL0QIHcJdOsEQ8wIIowc"
    },
    {
        "title": "Louis Vuitton Onthego Empreinte PM Black",
        "price": "$3,568.00",
        "image": "https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcTTFAXAgTlbBErj8kGSJaZlvTWabOLIhvd-YJDCNrJ8Ls05VdXaz2sClNaCQVaDJk0lzcO-RTspO0t1gFJqDayo6Y45dhKpxkTvXDlqqFjIkUJe_cweXqLgeBs&usqp=CAE",
        "retailer": "StockX",
        "product_url": "https://www.google.com/shopping/product/7199001631589324220?q=louis+vuitton+bags&num=20&udm=28&prds=eto:16054018195505784914_0,pid:14263685043257081273&sa=X&ved=0ahUKEwjOzJ7ex9uHAxUvL0QIHcJdOsEQ8wIIsAc"
    },
    {
        "title": "Louis Vuitton Graceful PM Beige Monogram",
        "price": "$1,760.00",
        "image": "https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcQCsPQIhiu7bBqMzKkmiFK1-VUHjK-RgE9gFSK57X9msI9NGef3Z5mIRn6Sc3I22QxgqZOWVDGd6RyrXzVMOuTG8WcVoUi2zGpRz0sOoOHPRfwK0QDlU3RbIA&usqp=CAE",
        "retailer": "louisvuitton.com",
        "product_url": "https://www.google.com/shopping/product/14078996663385130630?q=louis+vuitton+bags&num=20&udm=28&prds=eto:4593674092438761338_0,pid:18331182635014054384&sa=X&ved=0ahUKEwjOzJ7ex9uHAxUvL0QIHcJdOsEQ8wIIvgc"
    },
    {
        "title": "Louis Vuitton Neverfull Monogram mm Cerise Lining",
        "price": "$2,041.00",
        "image": "https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcR6-Imgjm6jD4hCOYHj6PrLzTAvenPZjAQl57txXE-RAetMXqDxZ8_sqb5OunjnjosKhHZiOZ61FJPh029Cs9v6pwo-_u03F6bt1sLOPKcSX6mLRW9UdNkg&usqp=CAE",
        "retailer": "ModeSens",
        "product_url": "https://www.google.com/shopping/product/3486536794175265554?q=louis+vuitton+bags&num=20&udm=28&prds=eto:3199910965274339290_0,pid:10652230344608010943,rsk:PC_17291330636138764321&sa=X&ved=0ahUKEwjOzJ7ex9uHAxUvL0QIHcJdOsEQ8wIIygc"
    },
    .... more
]

In the next sections, we’ll explore how to scrape individual Google Shopping product pages for more detailed information.

Google Shopping Product Page Structure

Once you’ve found products on the Google Shopping SERP, you can drill down into individual product pages to get more info. Understanding the structure of these pages is key to getting the most value.

Key Elements of a Google Shopping Product Page

Google Shopping Product Page
  1. Product Title and Description

The product title and description give you the main features and benefits of the product.

  1. Price and Availability

Detailed pricing, including any discounts and availability status, shows if it’s in stock or sold out.

  1. Images and Videos

Images and videos show the product from different angles so you can see what it looks like.

  1. Customer Reviews and Ratings

Reviews and ratings give you an idea of customer satisfaction and product performance so you can gauge quality.

  1. Specifications and Features

Specifications like size, color, and material help you make an informed decision.

  1. Retailer Information

Information about the retailer, including store name and contact info, so you can see who sells the product and may include shipping and return policies.

In the next section, we’ll write a scraper for Google Shopping product pages and save the scraped data to a JSON file.

Scraping Google Shopping Product Page

In this section, we will walk you through scraping individual Google Shopping product pages. This includes inspecting the HTML, writing a scraper and saving the extracted data to a JSON file.

Inspecting the HTML Structure

Before writing your scraper, use the browser’s Developer Tools to inspect the HTML structure of a Google Shopping product page.

Google Shopping Product Page Inspect
  • Right-click on a product listing and select “Inspect” to open the Developer Tools.
  • Hover over elements in the “Elements” tab to see which part of the page they correspond to.
  • Identify the tags and classes containing the data you want to extract, such as product titles, prices, and reviews.

Writing Google Shopping Product Page Scraper

To scrape a Google Shopping product page, we’ll use the Crawlbase Crawling API to retrieve the HTML content. Here’s how you can set up the scraper:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup
import json

# Initialize CrawlingAPI with your access token
crawling_api = CrawlingAPI({'token': 'CRAWLBASE_TOKEN'})

# Set options for the request
options = {
    'country': 'FR',  # Adjust the country code as needed
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
}

# Function to scrape a Google Shopping product page
def scrape_product_page(url):
    # Fetch the page using Crawlbase API
    response = crawling_api.get(url, options)
    if response['headers']['pc_status'] == '200':
        html_content = response['body'].decode('utf-8')
        soup = BeautifulSoup(html_content, 'html.parser')

        # Extract product details
        title = soup.select_one('.sh-np__product-title').get_text(strip=True) if soup.select_one('.sh-np__product-title') else None
        price = soup.select_one('.T14wmb').get_text(strip=True) if soup.select_one('.T14wmb') else None
        description = soup.select_one('.sh-np__description').get_text(strip=True) if soup.select_one('.sh-np__description') else None
        images = [img['src'] for img in soup.select('.sh-div__image img')]

        product_details = {
            'title': title,
            'price': price,
            'description': description,
            'images': images
        }
        return product_details
    else:
        print(f"Failed to fetch the page. Status code: {response['headers']['pc_status']}")
        return None

# Example URL for a product page
product_url = 'https://www.google.com/shopping/product/10571198764600207275'
product_details = scrape_product_page(product_url)

Saving Data to JSON File

Once you have extracted the product data, you can save it to a JSON file for analysis or further processing:

1
2
3
4
5
6
7
def save_to_json(data, filename='product_details.json'):
    with open(filename, 'w', encoding='utf-8') as f:
        json.dump(data, f, ensure_ascii=False, indent=4)

# Save product details to JSON
if product_details:
    save_to_json(product_details)

Complete Code

Below is the complete code to scrape a Google Shopping product page and save the data to a JSON file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup
import json

# Initialize CrawlingAPI with your access token
crawling_api = CrawlingAPI({'token': 'CRAWLBASE_TOKEN'})

options = {
    'country': 'FR',
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
}

def scrape_product_page(url):
    response = crawling_api.get(url, options)
    if response['headers']['pc_status'] == '200':
        html_content = response['body'].decode('utf-8')
        soup = BeautifulSoup(html_content, 'html.parser')

        title = soup.select_one('span.sh-t__title-pdp.sh-t__title').text.strip() if soup.select_one('span.sh-t__title-pdp.sh-t__title') else None
        price = soup.select_one('span.g9WBQb').text.strip() if soup.select_one('span.g9WBQb') else None
        description = soup.select_one('p.sh-ds__desc').text.strip() if soup.select_one('p.sh-ds__desc') else None
        images = [img['src'] for img in soup.select('div.main-image > img')]8.

        product_details = {
            'title': title,
            'price': price,
            'description': description,
            'images': images
        }
        return product_details
    else:
        print(f"Failed to fetch the page. Status code: {response['headers']['pc_status']}")
        return None

def save_to_json(data, filename='product_details.json'):
    with open(filename, 'w', encoding='utf-8') as f:
        json.dump(data, f, ensure_ascii=False, indent=4)

product_url = 'https://www.google.com/shopping/product/10571198764600207275'
product_details = scrape_product_page(product_url)

if product_details:
    save_to_json(product_details)

Example Output:

1
2
3
4
5
6
7
8
9
{
  "title": "Louis Vuitton Speedy Bandouliere 25 Black",
  "price": "$4,059.00",
  "description": "Please Note: This item comes with a dust bag, the box is not required.",
  "images": [
    "https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcQAipo2LXesayTJ_iC-3a6Z8JfDW4gvn11c26qcNRRBBTzdIpCmjW98JMysQ6W3iEsubeMIqem0ta5wT6-Q4LgXxSG9OG86BgaHrO4FoiD9WUlFv3ks3JjKzw&usqp=CAY",
    "https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcTWWrHYy89sfQwWynpx8_4qBe4hMMyEBOVsy8F_szfqkXJaB5DLzqNwJ2NaPfyalXnGQfc2DgOR3nOYoAIH-K_fKlKJNhGC&usqp=CAY"
  ]
}

In the next section, we’ll wrap up our discussion with final thoughts on scraping Google Shopping.

Scrape Google Shopping with Crawlbase

Scraping data from Google Shopping helps you understand product trends, prices, and what customers think. Using the Crawlbase Crawling API can help you avoid problems like IP blocking and content that change often, making data collection easier. By using Crawlbase to get the data, BeautifulSoup to read the HTML, and JSON to store the data, you can effectively gather detailed and valuable information.

As you implement these techniques, remember to adhere to ethical guidelines and legal standards to ensure your data collection practices are responsible and respectful.

If you’re interested in exploring scraping from other e-commerce platforms, feel free to explore the following comprehensive guides.

📜 How to Scrape Amazon
📜 How to scrape Walmart
📜 How to Scrape AliExpress
📜 How to Scrape Flipkart
📜 How to Scrape Etsy

If you have any questions or feedback, our support team is always available to assist you on your web scraping journey. Thank you for following this guide. Happy scraping!

Frequently Asked Questions

Scraping Google Shopping data can be legal, but it’s essential to follow the website’s terms of service, and many users find that with respect and caution, it can be done. If unsure, seek professional legal advice. Using official APIs when available is also an excellent way to get data without legal issues. Always scrape responsibly and within the guidelines.

Q. What data can I extract from Google Shopping product pages?

When scraping Google Shopping product pages, you can extract the following data points. Product title to identify the product, prices to show current price and discount, and description to show product features. You can also get images for visual representation, ratings and reviews for customer feedback, and specifications like size and color for technical details. This data is useful for market analysis, price comparison, and customer opinions.

Q. How can I handle websites that block or limit scraping attempts?

Websites block scraping by using IP blocking, rate limiting, and CAPTCHAs. To handle these issues use IP rotation services like Crawlbase Crawling API to avoid IP blocks. Rotate user agents to mimic different browsers and reduce detection risk. Implement request throttling to space out your requests and avoid rate limits. For CAPTCHA bypass, some APIs, including Crawlbase can help you to overcome these hurdles and get continuous data extraction.

Q. What should I do if the product page structure changes?

If Google Shopping product page structure changes, you will need to update your scraping code to adapt to the new layout. Here’s how to

  • Regular Monitoring: Monitor the product page regularly to detect any updates or changes in the HTML structure.
  • Update Selectors: Update your scraping code to reflect new tags, classes or IDs used on the page.
  • Test Scrapers: Test your updated code to make sure it extracts the required data with the new structure.
  • Handle Exceptions: Implement error handling in your code to handle scenarios where expected elements are missing or altered. Be proactive and adapt to changes to get accurate data extraction.