How to Analyze Competitor Google Adsfrom SERP ads to ad intelligence

Google Ads is a competitive space where you pay for visibility, and the paid results at the top of a search page tell you exactly which competitors are willing to bid for each query. Those ad slots are public: anyone can run a search and read the headlines, display URLs, and copy your rivals put in front of the same audience you are chasing. Captured at scale across your target keywords, that becomes ad intelligence you can act on.

This guide shows you how to analyze competitor Google Ads by capturing the paid results on a Google SERP through the Crawling API for a list of keywords, parsing each ad's title, display URL, and copy, and then working out which competitors bid on which terms. We keep the whole walkthrough scoped to public ad data that any searcher can see, and the legality section near the end is not boilerplate, so read it before you point this at real volume.

What you will build

A Python script that takes a list of target keywords, runs each as a public Google search through the Crawling API, and extracts a structured record for every paid ad on the page. The Crawling API's built-in Google SERP scraper returns the ads already parsed, so for each ad you get these fields:

• Position the rank of the ad among the paid results, counted from the top.
• Title the ad headline, the bold clickable text the advertiser wrote.
• Display URL the visible destination shown under the headline, which is the competitor's brand.
• Description the ad copy, the one or two lines of messaging and call to action.
• Site links the extra deep links some advertisers attach below the main ad.

From there you roll the per-keyword results up into a simple competitor view: which brands show ads on which keywords, and how often.

Why a plain request fails on Google

If you fire a bare HTTP request at a Google search URL from a script, you rarely get the page you see in your own browser. Google renders much of the SERP with JavaScript and watches closely for automated traffic. A request that does not look like a real browser gets a consent wall, a CAPTCHA, or an outright block long before it reaches the ad markup, and the ad slots in particular are personalised and rendered late, so a raw fetch often returns a page with no ads on it at all.

So a working Google Ads scraper needs two things in one request: an IP the platform reads as a real visitor, and a browser that renders the page. You can assemble that yourself with a headless browser plus a pool of rotating residential proxies, but keeping those healthy is most of the work. The Crawling API folds both into a single call: you send it the search URL, it fetches from a trusted residential IP and renders the page, and its Google SERP scraper returns the ads already parsed into structured JSON.

Key data points to extract

Before the code, it is worth being clear about what each field tells you, because the analysis is only as good as the fields you collect.

• Ad headlines and descriptions. Competitor ad copy shows what messaging they think works. Look for repeated language, emotional triggers, and the calls to action that earn clicks.
• Targeted keywords. If a competitor's ad shows for a keyword, they are bidding on it, which is a strong signal that the term converts for them.
• Ad placement and ranking. An advertiser that consistently holds the top paid slot is bidding aggressively or running high-quality ads, both worth understanding.
• Display and landing URLs. The display URL identifies the competitor; the destination shows how they convert traffic, from page design to offer.
• Ad extensions. Site links, callouts, and structured snippets make an ad larger and more compelling. Note who uses them and how.

Prerequisites

You need a few things in place before writing any code. None of them take long.

Basic Python. You should be comfortable writing and running a Python script and installing packages with pip. If parsing JSON output is new to you, our guide to scraping websites with Python covers the basics this tutorial assumes.

Python 3.8 or later. Confirm your version with python --version. If you do not have it, install it from python.org or through a distribution like Anaconda.

A Crawlbase account and token. Sign up, open your dashboard, and copy your request token. Your first 1,000 requests are free. The SERP scraper renders the page, so use your normal (JavaScript) request token. Treat the token like a password: it authenticates your requests, so keep it out of version control.

Set up the project

Install the official Crawlbase Python library, which wraps the Crawling API and its SERP scraper in a small client.

python --version

python -m venv ads_env
source ads_env/bin/activate

pip install crawlbase

On Windows, activate the environment with ads_env\Scripts\activate instead of the source line. The crawlbase package gives you a CrawlingAPI client that sends the request, applies the Google SERP scraper, and returns the parsed response, so you do not write any HTML parsing for the ads yourself.

Step 1: Fetch a SERP and pull the ads

Start by getting the ads for a single keyword. Write a scrape_google_ads() function that builds the search URL, sends it through the Crawling API with the google-serp scraper, checks the status, and returns the parsed ads array.

import json
from urllib.parse import quote
from crawlbase import CrawlingAPI

API_TOKEN = "YOUR_CRAWLBASE_TOKEN"  # replace with your token
crawling_api = CrawlingAPI({"token": API_TOKEN})

def scrape_google_ads(query):
    url = f"https://www.google.com/search?q={quote(query)}"
    options = {"scraper": "google-serp"}
    response = crawling_api.get(url, options)

    if response["headers"]["pc_status"] != "200":
        print(f"Failed to fetch '{query}'")
        return []

    parsed = json.loads(response["body"].decode("latin1"))
    return parsed.get("body", {}).get("ads", [])

if __name__ == "__main__":
    ads = scrape_google_ads("project management software")
    print(f"Found {len(ads)} ads")
    print(json.dumps(ads[:1], indent=2, ensure_ascii=False))

The SERP scraper returns a JSON envelope. You read two things: response["headers"]["pc_status"] is the status of the underlying Google fetch, and response["body"] is the parsed payload, which you decode and load with json.loads. Guarding on pc_status means a block or consent wall surfaces as a clear failure instead of feeding an empty page into your analysis. The ads live under body.ads, already split into individual records, so a single keyword like "project management software" comes back as a list you can inspect. Run the script and you should see a count and the first ad printed, which confirms the fetch and parse work before you scale up.

Step 2: Normalise the ad fields

The parsed ads give you everything you need, but it pays to flatten each one into a consistent record so the analysis step does not have to reach into nested keys. Write a small parse_ads() function that pulls the title, display URL, description, destination, and any site links from each ad, and stamps the keyword that produced it.

def parse_ads(query, ads):
    records = []
    for ad in ads:
        site_links = [
            {
                "title": link.get("title"),
                "url": link.get("url"),
            }
            for link in ad.get("siteLinks", [])
        ]
        records.append({
            "keyword": query,
            "position": ad.get("position"),
            "title": ad.get("title"),
            "displayUrl": ad.get("displayUrl") or ad.get("url"),
            "description": ad.get("description"),
            "destination": ad.get("url"),
            "siteLinks": site_links,
        })
    return records

Each record now carries the keyword it came from, the ad's position, the title headline, the displayUrl (the visible brand, falling back to the destination if the display field is absent), the description copy, the real destination URL, and a flattened list of any siteLinks. Stamping the keyword on every ad is what makes the later analysis possible: it lets you group ads by competitor across many keywords without losing track of which term each one appeared on.

Step 3: Run a keyword list and analyze competitors

Now wire it together. Loop over your target keywords, scrape and parse the ads for each, collect every record, and then roll the records up into a competitor view. Pace the loop with a short sleep so you are not firing requests back to back.

import json
import time
from urllib.parse import quote, urlparse
from collections import defaultdict
from crawlbase import CrawlingAPI

API_TOKEN = "YOUR_CRAWLBASE_TOKEN"
crawling_api = CrawlingAPI({"token": API_TOKEN})

KEYWORDS = [
    "project management software",
    "crm for small business",
    "time tracking app",
]

def scrape_google_ads(query):
    url = f"https://www.google.com/search?q={quote(query)}"
    response = crawling_api.get(url, {"scraper": "google-serp"})
    if response["headers"]["pc_status"] != "200":
        print(f"Failed to fetch '{query}'")
        return []
    parsed = json.loads(response["body"].decode("latin1"))
    return parsed.get("body", {}).get("ads", [])

def parse_ads(query, ads):
    return [
        {
            "keyword": query,
            "position": ad.get("position"),
            "title": ad.get("title"),
            "displayUrl": ad.get("displayUrl") or ad.get("url"),
            "description": ad.get("description"),
            "destination": ad.get("url"),
        }
        for ad in ads
    ]

def domain_of(record):
    target = record["destination"] or record["displayUrl"] or ""
    return urlparse(target).netloc or target

def main():
    all_ads = []
    for keyword in KEYWORDS:
        ads = scrape_google_ads(keyword)
        all_ads.extend(parse_ads(keyword, ads))
        print(f"{keyword}: {len(ads)} ads")
        time.sleep(2)

    competitors = defaultdict(list)
    for record in all_ads:
        competitors[domain_of(record)].append(record["keyword"])

    report = {
        domain: {
            "adCount": len(keywords),
            "keywords": sorted(set(keywords)),
        }
        for domain, keywords in competitors.items()
    }

    with open("competitor_ads.json", "w", encoding="utf-8") as f:
        json.dump({"ads": all_ads, "competitors": report},
                  f, ensure_ascii=False, indent=2)
    print(f"Saved {len(all_ads)} ads from {len(report)} competitors")

if __name__ == "__main__":
    main()

Run the full script with python competitor_ads.py. It scrapes the ads for each keyword in KEYWORDS, flattens them into records, then groups every record by its competitor domain with defaultdict to build a report of how many ads each brand ran and across which keywords. The output goes to competitor_ads.json with both the raw ad records and the rolled-up competitor view. Add more terms to KEYWORDS and the same two functions handle the rest; the analysis grows with your list.

What the output looks like

You get a structured object with the parsed ads and a competitor summary, ready to load into a spreadsheet, a notebook, or a database for deeper analysis.

{
  "ads": [
    {
      "keyword": "project management software",
      "position": 1,
      "title": "Project Management Tool | Plan, Track and Ship Faster",
      "displayUrl": "https://www.example-pm.com/",
      "description": "Run every project in one place. Free 14-day trial, no card required.",
      "destination": "https://www.example-pm.com/trial"
    }
  ],
  "competitors": {
    "www.example-pm.com": {
      "adCount": 2,
      "keywords": ["crm for small business", "project management software"]
    }
  }
}

The competitors block is where the intelligence lives: it tells you at a glance which brands bid on which of your terms and how many of your tracked keywords each one appears on. A competitor showing up on several keywords is investing broadly; one that owns the top position on your highest-value term is worth a closer look at their copy and landing page.

Scaling across keywords and locations

One small keyword list is a demo; a real job tracks dozens of terms, often across regions, and reruns on a schedule to catch new entrants. The shape stays the same: build each search URL, fetch it through the Crawling API with the SERP scraper, and parse the ads. A few additions make it production-ready.

• Localise the SERP. Ad auctions differ by country, so add &gl=us and &hl=en (or another country and language) to the search URL to capture the ads a searcher in that market would see.
• Pace the loop. Keep the short sleep between requests rather than firing them back to back. Spread a large keyword list out instead of hammering Google in a tight loop.
• Schedule reruns. Ad copy and bidders change week to week, so a weekly run turns a one-off snapshot into a trend you can track over time.

Crawlbase serves up to 20 requests per second by default, which is ample for a paced keyword tracker, and any 5XX response from the API is free of charge, so retrying a blocked URL costs you nothing. If you would rather route your own traffic through a rotating pool, the Smart AI Proxy gives you the same residential rotation as a drop-in endpoint. For the parsing-and-blocking playbook around Google specifically, see how to rotate proxies for scraping Google search results and how to bypass CAPTCHA while scraping Google.

Combine scraping with the official transparency source

Scraping the live SERP tells you what is running right now for your exact keywords, which is the data you cannot get any other way. To round out the picture, pair it with Google's own free tool, the Google Ads Transparency Center. It lets you look up an advertiser and see the ads they are running across Google's network, the formats they use, and the regions they target. It is an official source, so it is the right place to study a known competitor's creative at depth.

The two approaches complement each other: scrape your keyword list to learn who competes on each term and how their copy reads in context, then open the Transparency Center on the brands that matter to see their wider campaign footprint. Beyond Google's own tool, paid platforms like SEMrush, SpyFu, and Ahrefs estimate ad spend and historical keyword coverage, which is useful for trend work but less precise than reading the live SERP yourself.

Is it legal to scrape Google Ads data?

Whether scraping Google Ads data is allowed depends on Google's terms of service, your jurisdiction, and what you do with the data. Google's terms place limits on automated access to its search service, so scraping can run against those terms regardless of how careful your tooling is. The ad slots themselves are public, anyone can see them by running the same search, but "public" is not the same as "unrestricted." None of the code here changes Google's terms; it only makes the technical part work. Read Google's terms and its robots.txt, and treat both as the boundary for what you collect.

A few lines worth holding to. Collect only public ad data: the titles, display URLs, descriptions, positions, and site links that any searcher can see on a results page without an account. Keep your request volume low enough that you are not straining Google's servers, and pace your crawl rather than running it flat out. Do not scrape anything behind a login, do not collect personal data, and do not republish a competitor's copyrighted ad creative as your own. You are gathering competitive intelligence to inform your strategy, not building a mirror of someone else's campaigns.

Where an official source exists, prefer it. Google's Ads Transparency Center is a sanctioned way to study a given advertiser's running ads, and it is the correct path when you want depth on a specific competitor rather than a live read of your own keyword list. This guide is deliberately scoped to public, visible ad results because that is the line that keeps the work defensible. If your project needs more than that, an official data agreement or one of the licensed ad-intelligence platforms is the right route, not a cleverer scraper.

Frequently Asked Questions (FAQs)

Why does a plain request return a Google page with no ads on it?

Google renders much of the SERP, including the ad slots, with JavaScript and personalises it per request, and it blocks traffic that does not look like a real browser. A bare HTTP call often gets a consent wall, a CAPTCHA, or a stripped page with no ads at all. Fetching through the Crawling API, which uses rotating residential IPs and renders the page, makes the request look like an ordinary visitor so the ads are actually present, and its SERP scraper returns them already parsed.

What ad fields can I extract from a Google SERP?

The Google SERP scraper returns an ads array where each ad carries a position, title (the headline), display URL, description (the copy), destination URL, and any site links. This tutorial flattens those into one record per ad and stamps the keyword that produced it, which is enough to analyze copy, placement, and which competitors bid on which terms.

How do I find which competitors bid on a keyword?

Run each target keyword as a search, collect the ads, and group the records by their competitor domain (derived from the display or destination URL). A brand that appears for a keyword is bidding on it. Roll those groups up across your whole keyword list and you get a per-competitor view of coverage, which is the core of the analysis in this guide.

Should I use the Google Ads Transparency Center instead of scraping?

Use both. The Transparency Center is Google's official tool for seeing the ads a given advertiser runs across the network, so it is ideal for studying a known competitor's creative in depth. Scraping the live SERP for your own keyword list tells you who competes on each term right now and how their copy reads in context, which the Transparency Center does not give you directly. They answer different questions.

How can I avoid getting blocked while scraping Google Ads?

Lean on rotating residential IPs so requests come from many real-user addresses, render the page so the ads load, and pace your requests rather than firing them in a tight loop. The Crawling API handles the rotation and rendering for you; if you roll your own stack, those are the parts to get right. Reading the status on every response also lets you back off the moment Google starts returning challenges.

Can I track ad changes over time?

Yes. Because each run writes a structured snapshot keyed on keyword and competitor domain, scheduling a weekly run and diffing the snapshots shows you new entrants, dropped advertisers, and changes in copy or position. For a broader use of this kind of competitive data, see our guide on using web scraping for price intelligence.

HR

Hassan Rehan

Software Engineer · Crawlbase

Software engineer at Crawlbase writing hands-on guides on rotating proxies, scraping, and the practical details of wiring proxies into real code.