On November 16, 2026, a request for the Google Trends Explore page for web scraping came back with HTTP 200. Crawlbase reported cb_status=200, and Google's own answer was recorded as original_status=200. The body said "Oops! There was a problem displaying this page" and asked the reader to try again. There was no keyword history in it.

The Explore Markdown path did that on all twenty first attempts. Nineteen bodies carried the Oops message and one carried a reCAPTCHA notice. A check that reads only the HTTP status would have accepted every one of them. A pipeline has to establish that the Trends data it asked for actually arrived before it counts the request as useful work.

This post follows the companion repository: request construction, response handling, parsers, content checks, a bounded retry, and the evidence the harness recorded. The Crawling API supplies the acquisition and the built-in extraction paths. The application decides which returned bodies meet its data requirements. The figures come from one region on one calendar day and are reported as observations, not as a benchmark.

The short version
  • HTTP 200 and cb_status=200 describe the document that arrived. For Google Trends, that document can be an error notice. Read the body before you accept the response.
  • The built-in scrapers passed most often: Trending Now 20/20 and Explore 18/20 with the JavaScript token. Explore as Markdown on the Normal token passed 0/20.
  • Retry on the verdict, never on the status code. One retry, both attempts kept, and a passing 200 is never retried.
Success is the shell, not the data. The status describes the document that arrived; the check has to read the body.

The diagram separates receiving a document from receiving the required data. It does not identify which browser request or widget failed inside Google Trends, and the run did not establish that either.

The test covers three URLs. Interest by subregion is part of the Explore response, so keyword history and regional interest do not need separate target pages in this implementation.

Surface URL in the test Data the project uses
Trending Now https://trends.google.com/trending?geo=US Trending queries, search-volume labels, growth, start time, status, and breakdown terms
Explore https://trends.google.com/trends/explore?q=web scraping&geo=US&date=today 12-m Interest over time, interest by subregion, and related queries
Trending RSS https://trends.google.com/trending/rss?geo=US Item title, approximate traffic, and publication date

Trending Now answers which searches are surging in the selected region; the saved scraper sample reflects the page's "Past 24 hours" filter. Explore answers how interest in one term varies over a chosen period. Google's Trending Now documentation lists RSS as an export option, and the feed is enough when its smaller set of fields covers the job.

Google's official Trends API alpha is a separate interface. Access was limited to selected testers at the time of the run, and the public overview did not list Trending Now data. It was not called in this project. The project also avoids the unofficial pytrends library, whose repository was archived on April 17, 2025, with a last push recorded as August 10, 2024. That maintenance status matters when a new pipeline picks a dependency.

Four surfaces, five paths, nine tested cells. The official API is shown for context and was not part of the test.

Make the request path explicit

The fetch-path module defines five request shapes. Every path goes through the Crawling API endpoint at https://api.crawlbase.com/, including the request for the RSS feed.

Path Parameters that distinguish it Token used in the matrix
Rendered HTML page_wait and ajax_wait=true JavaScript token
Built-in scraper scraper=google-trends or scraper=google-trends-explore Normal for Trending Now; JavaScript for Explore
Markdown format=md Normal token
No-wait HTML Target URL with no wait parameters JavaScript token
RSS fetch The RSS target URL Normal token

The no-wait control still renders JavaScript. Its only difference from the rendered path is the absence of the two explicit wait parameters. In the same way, the Markdown results describe the Normal-token Markdown path that was actually tested. Every comparison in this post names its token and request settings.

The request builder keeps token values out of parameter preparation. For scraper requests it returns no preferred token kind; the caller supplies the choice from probe-config.json. The client then reads the matching secret and passes the parameters to requests.Session.get() with a 180-second timeout:

python
def prepare(
    path: str,
    url: str,
    *,
    page_wait: int | None = None,
    ajax_wait: bool | None = None,
    scraper: str | None = None,
    store: bool = False,
) -> tuple[str | None, dict]:
    """Return (token_kind, params) for a path.

    token_kind is "js", "normal", or None. None means the caller chooses
    the token (the built-in scrapers do not name one).
    """
    params = {"url": url}
    if store:
        params["store"] = "true"
    if path == "rendered":
        params["page_wait"] = str(page_wait)
        params["ajax_wait"] = "true" if ajax_wait else "false"
        return "js", params
    if path == "scraper":
        params["scraper"] = scraper or ""
        return None, params
    if path == "markdown":
        params["format"] = "md"
        return "normal", params
    if path == "nowait":
        return "js", params
    if path == "rss":
        return "normal", params
    raise ValueError(f"unknown path {path}")

Rendered Trending Now uses page_wait=5000 and ajax_wait=true. Rendered Explore uses page_wait=8000 and ajax_wait=true. page_wait is a fixed delay in milliseconds after page load. ajax_wait=true waits for network activity to settle before the page is captured, and adds no fixed delay of its own.

Keep the probe's token decision inspectable

The probe makes thirteen target calls. It tries both token types for each built-in scraper, the HTML and Markdown paths, RSS, a separate comparison query, and one RSS request with storage enabled. It also records account summaries before and after, with at least five minutes between the two account calls.

The probe's raw bodies are not checked in, so the finding lives in the configuration the matrix reads: empty chart objects with the Normal token and 53 timeline points with the JavaScript token. The matrix therefore uses the JavaScript token for Explore:

json
{
  "probed_on": "2026-11-16",
  "trending_scraper_token": "normal",
  "explore_scraper_token": "js",
  "trending_page_wait": 5000,
  "trending_ajax_wait": true,
  "explore_page_wait": 8000,
  "explore_ajax_wait": true,
  "store": false,
  "store_reason": "re-crawling the same URL with store=true replaces the stored copy, so it cannot keep 20 runs",
  "explore_scraper_token_reason": "On the 16 Nov 2026 probe the normal token returned empty chart objects. The JS token returned 53 timeline points."

The probe log still contains verdicts generated before the envelope handling and the content checks were corrected, so those original verdicts do not reflect the final rules. The configuration is part of the reproducible experiment. A fresh probe can replace it. Inspect the token selections before starting a matrix, because the matrix never switches tokens or picks another fetch path after a failed response.

Read the response layers before parsing the data

The fetch executor records the outer status, the Crawlbase status, the target status, the final URL, elapsed time, body size, and the domain complexity. A transport exception becomes a result with no HTTP response and an empty body, with token values scrubbed from the exception text.

Signal What it establishes in this project
Outer HTTP status Whether a response arrived from Crawlbase, and with which status
cb_status Crawlbase's crawl outcome; the application requires 200 before it accepts content
original_status Google's reported HTTP status, kept in the CSV
Application verdict Whether the body satisfies the checks for the requested surface and path

Scraper responses need one extra step. In the observed JSON envelope, cb_status, original_status, domain_complexity, and url surround an inner body object. The executor unwraps that envelope before validation. Looking only at headers would miss the metadata it carries, which is what went wrong in the first probe:

python
def unwrap_envelope(body: str):
    """Scrapers return a JSON envelope. The fields we check are under body."""
    text = (body or "").lstrip("\ufeff").strip()
    if not text.startswith("{"):
        return body, None, None, None, None
    try:
        data = json.loads(text)
    except json.JSONDecodeError:
        return body, None, None, None, None
    if not isinstance(data, dict) or "body" not in data or "cb_status" not in data:
        return body, None, None, None, None
    inner = data.get("body")
    if isinstance(inner, (dict, list)):
        inner_text = json.dumps(inner)
    elif inner is None:
        inner_text = ""
    else:
        inner_text = str(inner)
    return (
        inner_text,
        data.get("cb_status"),
        data.get("original_status"),
        data.get("domain_complexity"),
        data.get("url"),
    )

The executor measures body_bytes from the original response text encoded as UTF-8, before unwrapping, so for scraper JSON the figure includes the envelope. It is not a compressed transfer size. Elapsed milliseconds cover the transport call; the 180-second value follows Requests' timeout semantics rather than a strict wall-clock deadline.

The first acceptance gate lives in assertions.py. It strips script and style blocks and tags, then searches the visible text for the configured markers: the Oops message, reCAPTCHA, unusual traffic, /sorry/, and g-recaptcha. Dropping scripts first stops a normal page's reCAPTCHA library reference from being read as a challenge, and the regression test test_recaptcha_script_on_a_data_page_is_not_a_challenge pins that behaviour:

python
def _visible_text(body: str) -> str:
    """Drop scripts and tags so a library URL is not mistaken for a challenge."""
    text = re.sub(r"(?is)<script\b[^>]*>.*?</script>", " ", body)
    text = re.sub(r"(?is)<style\b[^>]*>.*?</style>", " ", text)
    text = re.sub(r"(?is)<[^>]+>", " ", text)
    return text


def challenge_reason(body: str) -> str | None:
    if not body:
        return None
    lowered = _visible_text(body).lower()
    for marker in CHALLENGE_MARKERS:
        if marker.lower() in lowered:
            return f"challenge marker: {marker}"
    return None


def gate(body: str | None, cb_status: int | None) -> Verdict | None:
    """Return a failing verdict, or None when the body may be parsed."""
    text = body or ""
    challenge = challenge_reason(text)
    if cb_status != 200:
        if challenge:
            return Verdict(False, f"cb_status {cb_status}, {challenge}")
        return Verdict(False, f"cb_status {cb_status}")
    if text.strip() == "":
        return Verdict(False, "empty body")
    if challenge:
        return Verdict(False, challenge)
    return None

The marker category groups notices the application cannot use. An Oops message on its own does not say why Google failed to display the page. And once the gate lets a body through to the parser, the required data still has to pass its own checks.

Parse the fields the job actually needs

The parsers turn each response format into the values the assertions expect. The check dispatcher picks the rules for the surface and path and adds the transport, language, and geography handling.

For Trending Now scraper JSON, the parser reads the structured trends array. For HTML and Markdown it first looks for visible English volume lines such as 20K+ searches and the titles that precede them, and it also reads HTML table rows and Markdown tables. The format differences are explicit: the structured scraper path checks more fields than the markup paths do.

Explore parsing looks for structured chart data first, including a JSON object embedded in the document. When those fields are absent it reads visible lines under English section headings. Timeline extraction from visible text expects a date label such as Nov 16, 2026 followed by an integer, so localized headings or dates can leave the parser with no timeline even when a complete localized page arrived.

Response Acceptance rules the application implements
Trending Now scraper JSON At least ten records; every record has position, title, searchVolume, increasePercentage, started, and status; trendsCount equals the array length
Trending Now HTML or Markdown At least ten extracted rows, each with a title and a volume label matching the expected pattern
Explore At least 40 timeline values; each integer-equivalent and within 0 to 100; the maximum is 100; the extracted subregion list is nonempty
Explore scraper geography A recognized United States label, or the literal geo=US in the returned URL
RSS At least five items, each with a title, approximate traffic, and publication date

The timeline assertion accepts integers, integer-valued floats, and integer strings, and rejects booleans. It does not validate dates, chronological continuity, the requested term, or every regional score:

python
def assert_timeline(values) -> Verdict:
    if not isinstance(values, list) or len(values) < TIMELINE_FLOOR:
        count = len(values) if isinstance(values, list) else 0
        return Verdict(
            False,
            f"timeline length {count} below {TIMELINE_FLOOR}",
            timeline_passed=False,
        )
    numbers = []
    for value in values:
        number = _as_int(value)
        if number is None:
            return Verdict(
                False,
                f"timeline value {value!r} is not an integer",
                timeline_passed=False,
            )
        if number < 0 or number > 100:
            return Verdict(
                False,
                f"timeline value {number} outside 0 to 100",
                timeline_passed=False,
            )
        numbers.append(number)
    if max(numbers) != 100:
        return Verdict(False, "timeline max is not 100", timeline_passed=False)
    return Verdict(True, "ok", timeline_passed=True)

These thresholds belong to this experiment's US query over today 12-m. Forty points is a minimum coverage rule, not a definition of a complete year. The peak-of-100 requirement fits the single-series chart this harness expects and should not become a claim that every Trends response without a 100 is a broken parse. The geography requirement applies to the Explore scraper path only; the HTML and Markdown paths do not enforce it.

Preserve the meaning of the numbers

Explore's timeline values are normalized interest scores. Google's explanation of Trends normalization describes relative search interest for a time and location, scaled to 0 to 100. A value of 50 is not fifty searches. A zero can mean low search volume; it is not automatically a missing observation. In this project an empty timeline is a failed extraction, while a timeline that contains zeros is checked as a series of values.

Trending Now's searchVolume is a different measurement. Labels such as 1M+ are bucketed search traffic, and increasePercentage is growth against a predicted baseline. The implementation keeps those strings as they are. It does not turn 1M+ into an exact count, convert relative started labels into timestamps, or derive absolute search counts from Explore scores.

The subregion parser keeps region names for the nonempty-list check. Passing that check does not verify every regional value or fetch every page of regions. The scraper returned the first five subregions the tested page displays, together with a total count and pagination text, and the project did not page through the rest.

The check, then the retry. Related queries that share no token with the term are recorded beside the pass, not used to fail a good timeline.

The diagram groups the checks by purpose. The functions linked above define the actual order and which checks apply to each response format.

Retry a failed verdict and keep both attempts

The harness receives three callables: fetch, check, and sleep. Its retry decision follows the verdict, and every attempt in a completed batch becomes its own record with the statuses, timing, body size, verdict, reason, and diagnostics:

python
def run_with_retry(fetch, check, sleep, *, surface, path, run, date=""):
    """Fetch and check. Sleep and try once more only if the check fails."""
    attempts = []
    for attempt in (1, 2):
        result = fetch()
        verdict = check(result)
        attempts.append(
            Attempt(
                surface=surface,
                path=path,
                run=run,
                attempt=attempt,
                date=date,
                latency_ms=result.latency_ms,
                body_bytes=result.body_bytes,
                credits=getattr(result, "credits", None),
                passed=verdict.passed,
                reason=verdict.reason,
                related_flagged=getattr(verdict, "related_flagged", None),
                related_share=getattr(verdict, "related_share", None),
                timeline_passed=getattr(verdict, "timeline_passed", None),
                http_status=getattr(result, "http_status", None),
                cb_status=getattr(result, "cb_status", None),
                original_status=getattr(result, "original_status", None),
                target_url=getattr(result, "target_url", ""),
                final_url=getattr(result, "final_url", None),
                my_location=getattr(verdict, "my_location", None),
                classic_banner=getattr(verdict, "classic_banner", None),
                selected_geo=getattr(verdict, "selected_geo", None),
                domain_complexity=getattr(result, "domain_complexity", None),
            )
        )
        if verdict.passed:
            break
        if attempt == 1:
            sleep(10)
    return attempts

A passing first attempt ends the loop. A failing first attempt earns one retry of the same configured request after ten seconds. Completed batches keep separate CSV rows for the first attempt and its retry, even when the retry passes.

The matrix runner visits the nine applicable cells in each round with one request in flight at a time, and waits three seconds after each cell's batch. When a cell needs a retry, the ten-second wait happens inside that batch, so the gaps are not a claim of exactly three seconds between every pair of calls.

The November 16 matrix made 221 target calls: 180 first attempts and 41 retries. Retries followed the rules in force during that run. Applying corrected checks later changed how saved responses were scored; it did not change which requests had been retried.

The summary groups records by surface and path and counts only attempt == 1 toward pass counts, median latency, and median body size. Those medians include passing and failing first attempts alike, so the results table does not blend initial reliability with recovery:

python
def summarise(attempts):
    """First-attempt pass rate, median latency, median body size, credit sum."""
    keys = sorted({(item.surface, item.path) for item in attempts})
    rows = []
    for surface, path in keys:
        first = [
            item
            for item in attempts
            if item.surface == surface and item.path == path and item.attempt == 1
        ]
        runs = len(first)
        passes = sum(1 for item in first if item.passed)
        known = [item.credits for item in first if item.credits is not None]
        rows.append(
            {
                "surface": surface,
                "path": path,
                "runs": runs,
                "passes": passes,
                "pass_rate": (passes / runs) if runs else None,
                "median_latency_ms": median([item.latency_ms for item in first]),
                "median_body_bytes": median([item.body_bytes for item in first]),
                "credits": sum(known) if known else None,
            }
        )
    return rows

What came back

The table reports the November 16, 2026 run with the corrected checks applied to its saved responses. Each row holds twenty first attempts. Explore used the fixed query web scraping, geo=US, and date=today 12-m; Trending Now and RSS used geo=US without a keyword.

Surface Path Pass, first attempt Pass, English pages Median latency (ms) Median body (bytes)
Trending Now Built-in scraper 20/20 20/20 10,607 6,864
Trending Now Markdown 20/20 n/a, no language tag 11,151 5,110
Trending Now Rendered HTML 18/20 18/18 26,077 2,351,950
Trending Now No-wait HTML 15/20 15/15 19,735 2,323,239
Explore Built-in scraper 18/20 14/16 26,029 4,075
Explore Markdown 0/20 n/a, no language tag 44,669 475
Explore Rendered HTML 17/20 17/17 27,493 705,602
Explore No-wait HTML 15/20 15/16 34,444 686,376
Trending RSS RSS fetch 20/20 n/a, XML 2,850 22,028

The English-pages column uses the HTML language tag for HTML responses and the language of the geography label for scraper JSON. It excludes identified non-English pages and the transport failure. No language classification was recorded for Markdown or RSS, so those cells read n/a. The first-attempt column counts all twenty attempts per path, including failures on localized pages.

First-attempt passes out of twenty per path, with the median body each path handed the parser. The two scrapers and RSS pass most often and return the least.

The original CSV keeps the verdicts used during the live run. The corrected pass counts and the timing and size figures can be recomputed from the rescored CSV. Medians are rounded to whole milliseconds and bytes. Credits were not measured.

The built-in Trending Now scraper passed 20 of 20 first attempts, with a median latency of 10,607 milliseconds and a median response of 6,864 bytes. This is the first record from the trimmed two-trend sample:

json
{
  "position": 1,
  "title": "steelers vs bengals",
  "searchVolume": "1M+",
  "searchVolumeText": "1M+ searches",
  "increasePercentage": "1,000%",
  "started": "19 hours ago",
  "status": "Active",
  "trendBreakdown": [
    "bengals vs steelers",
    "joe burrow",
    "ja'marr chase"
  ],
  "trendBreakdownMoreCount": 78
},

The sample keeps the original trendsCount=25 and pagination metadata while retaining only two records for illustration. It is a shortened live sample, not a complete body to submit to the ten-record assertion.

Rendered Trending Now HTML passed 18 of 20, with a median body of 2,351,950 bytes. The rescored CSV identifies both failures as Portuguese pages the English parser could not read; French appears among the no-wait Trending Now failures. The English-only rendered subset passed 18 of 18.

Trending Now Markdown also passed 20 of 20, with a median body of 5,110 bytes, by matching the required English title-and-volume patterns. Passing that check does not mean the six fields validated on scraper JSON were validated on Markdown.

Explore: a 200 that fails every content check

Explore Markdown passed 0 of 20 first attempts. All twenty recorded HTTP 200, cb_status=200, and original_status=200. Nineteen contained the Oops message and one the reCAPTCHA notice. Across first attempts and retries, the Oops message appeared in 36 of the forty saved Explore Markdown bodies. This is the saved error sample, with the cookie banner line removed:

text
Oops! There was a problem displaying this page.

 Please try again

Google apps

Please upgrade to a [supported browser](https://support.google.com/recaptcha/?hl=en#6223828) to get a reCAPTCHA challenge.

[Why is this happening to me?](https://support.google.com/recaptcha#6262736)

The path returned a small, parseable document, with a median body of 475 bytes, and none of the required keyword data.

Rendered Explore HTML passed 17 of 20, with a median latency of 27,493 milliseconds and a median body of 705,602 bytes. Its three failures were non-English pages; the English-only subset passed 17 of 17. The no-wait Explore control passed 15 of 20: three non-English pages, one transport failure with no response body, and one English page with an empty extracted timeline. The comparison supports the recorded wait configuration for this case. Twenty attempts per path do not establish that a wait fixes every empty chart.

Explore scraper: the original score and the corrected score

The Explore scraper originally passed 14 of 20 first attempts. Four more responses had timelines and subregions but failed the original geography rule because the United States label was localized: Spanish or Portuguese, Russian, or Dutch. The corrected checks accept recognized localized labels and fall back to geo=US in the response URL for the scraper path.

Rescoring the same bodies raised the first-attempt count to 18 of 20, and one Polish-label retry moved from fail to pass. The rescored report records 23 changed verdicts overall: five pass/fail changes and eighteen reason changes, with four of the five pass/fail changes on first attempts.

The corrected scraper's two remaining first-attempt failures were empty timelines on English pages. Its English-label subset was 14 of 16 under both rule versions. Its median response was 4,075 bytes against 705,602 bytes for rendered Explore HTML. The scraper cut the response the application had to handle by two orders of magnitude while keeping the timeline and the displayed subregions.

Language explains much of the raw-versus-filtered gap

Seventeen first attempts were identified as localized pages or localized scraper labels. Thirteen failed the English parser, across Russian, Turkish, Slovak, Hebrew, Ukrainian, French, Portuguese, Chinese, and Persian pages. The other four were the Explore scraper responses accepted after the geography correction. The dispatcher labels those parse failures so a localized full page is not reported as an empty widget:

python
def check_result(surface: str, path: str, result, term: str = TERM):
    verdict = _check(surface, path, result, term)
    if verdict.passed:
        return verdict
    lang = page_lang(result.body or "")
    if (
        lang
        and not lang.lower().startswith("en")
        and surface in ("trending", "explore")
        and verdict.reason.startswith(LOCALE_SENSITIVE)
    ):
        return dataclasses.replace(
            verdict,
            reason=f"non-English page ({lang}), localized labels not read: {verdict.reason}",
        )
    return verdict

That label is a parser limitation observed against localized content. It does not establish that Google's widgets were empty, and it does not make those responses usable by the current application, so they stay failures in the raw pass count.

No language setting was sent. Variation with the request route is a plausible explanation for the changing page language, but the run does not prove that mechanism. Adding hl=en-US is a useful next experiment, not a control included in these results. The post reports one region and one calendar day, with no second-day or language-pinned matrix.

Inspect diagnostics that a passing timeline can hide

Structural validation catches the error pages and the empty timelines in this run. It also leaves room to record suspicious fields without discarding a usable part of the response.

Across the 59 first attempts that recorded the related-query flag, none had a related term that shared a token with web scraping, and that count includes responses whose related list was empty. The report lowercases the term, extracts alphanumeric tokens of at least three characters, adds a simple stem for qualifying -ing words, and looks for those tokens as substrings in the related names:

python
def term_tokens(term: str) -> list[str]:
    tokens = []
    for word in re.findall(r"[a-z0-9]+", term.lower()):
        if len(word) < 3:
            continue
        tokens.append(word)
        if word.endswith("ing") and len(word) > 5:
            stem = word[:-3]
            if len(stem) >= 3:
                tokens.append(stem)
    return tokens


def related_report(term: str, names) -> tuple[bool, float]:
    """Return (flagged, share). Flagged when no query shares a term token."""
    queries = [name for name in (names or []) if name]
    if not queries:
        return True, 0.0
    tokens = term_tokens(term)
    hits = 0
    for name in queries:
        lowered = str(name).lower()
        if any(token in lowered for token in tokens):
            hits += 1
    share = hits / len(queries)
    return hits == 0, share

An empty list is flagged too, so related_flagged=True on its own is not proof that a nonempty list arrived. This is a diagnostic heuristic, not a semantic relevance model, and the application deliberately leaves the timeline verdict intact when the related list is flagged. The observed unrelated terms justify checking query binding before related queries go downstream; this run did not establish their cause.

"My location" differed from the selected search region

Trending Now HTML exposed fifteen different countries through its "My location" label across 33 first-attempt rows. India appeared eight times; Vietnam, Indonesia, and Lebanon three times each. The selected trend region stayed United States in all 33 rows. The parser records my_location separately from selected_geo. Use the selected Trends geography to identify the data's scope. That "My location" follows the request route is an inference from these observations, not a documented guarantee from Google.

Classic Explore displayed an expiry banner

The banner "Classic Explore is expiring soon" appeared on seventeen of twenty rendered Explore pages and sixteen of twenty no-wait pages. The built-in scraper's JSON did not include it. classic_banner records its presence without using it as a pass condition. The banner gives no migration date, but it is an operational reason to rerun representative checks when the Explore interface changes, especially for parsers tied to visible headings and date formats.

Match the method to the job

The recorded run supports these starting points:

Job Path to use Evidence, November 16, 2026
Daily trend feed Trending Now built-in scraper, or RSS when title, approximate traffic, and publication date are enough Scraper 20/20, median 10,607 ms; RSS 20/20, median 2,850 ms
Keyword history Explore built-in scraper with the JavaScript token 18/20 under the corrected geography rule; two empty timelines
Regional breakdown Explore built-in scraper's subregion list Returned in the same response; the first five displayed subregions, not the full list
Agent tool call Trending Now built-in scraper as JSON 20/20; structured fields can be checked before data returns to the caller
Which path for which job. Rendered Explore stays a separately implemented option; the harness never invokes it as an automatic fallback.

Explore Markdown's 0/20 result applies to the tested Normal-token path. These results are a basis for choosing a configuration to verify on your own target query, not a substitute for that verification.

Run the implementation and verify what changed

The complete package imports the three step modules through load_steps.py, so the request shapes, assertions, and retry function quoted above are the code the CLI runs. The final suite has thirty tests covering assertions, parser behaviour, envelope handling, token redaction, retries, the billable-response cap, statistics, and CLI execution. Its parity test exercises the shared harness summary; it does not revalidate the live Google responses.

The requirements declare requests>=2.32 and pytest>=8 without pinning exact versions. The recorded run used Python 3.13. Copy the environment template to .env inside final/ and set CRAWLBASE_TOKEN and CRAWLBASE_JS_TOKEN from your Crawlbase account. The file is gitignored. The client adds the secret only at the HTTP boundary, its log records the token kind, and saved bodies and exception messages are scrubbed for the configured secrets. From the repository root:

bash
cd final
cp .env.example .env          # add CRAWLBASE_TOKEN and CRAWLBASE_JS_TOKEN
python -m pytest -q
python -m trends_probe --offline --date 2026-11-16
python -m trends_probe --matrix --runs 20 --max-billable 400 --date 2026-11-16 --geo US

The pytest and offline commands make no live target calls. The matrix command makes live calls and consumes credits. Its --date argument labels the output; it does not request a historical snapshot, and today 12-m is resolved when Google serves the request. The README covers setup on Windows, Linux, and macOS.

Keep fixtures and live evidence separate

The offline command uses synthetic fixtures: a minimal timeline, example trends, RSS items, an Oops body, and an empty response. Its summary identifies itself as a fixture run. Those results demonstrate program behaviour and must not be counted as additional live attempts.

For live runs, the CLI saves a scrubbed body for each call under results/raw/matrix/ and checkpoints completed batches to CSV. The public repository includes both recorded CSVs and the trimmed live samples; the full raw-body directory is gitignored.

--report rebuilds the summary from recorded verdicts. --rescore needs the matching local raw bodies, reapplies the current checks, and writes a separate -rescored CSV and Markdown report without new target requests. Preserve each run's raw directory with its CSV before another matrix: body filenames restart their call counter, and rescoring pairs rows with files in call order after checking their surface and path.

The matrix uses store=False and never reuses a previous successful body as a fallback. One probe request exercised store=true with Crawlbase Cloud Storage. Its log recorded a storage_url header and no separate rid; the identifier appears only inside that URL's query string. The checked-in config notes that storing the same URL again replaces the stored copy. The run therefore kept individual local bodies for comparison. The Crawling API reference documents an rid for stored requests; this implementation does not depend on that identifier.

Keep request counts separate from credit consumption

The matrix counted 220 billable responses against its configured cap of 400. The cap counts responses, not credits. The local classifier looks mainly at cb_status:

python
BILLABLE = {200, 207}


class BudgetStop(Exception):
    """The billable cap was reached before the matrix finished."""


def is_billable(result) -> bool:
    if result.cb_status in BILLABLE:
        return True
    if result.cb_status is None and result.http_status == 200 and not result.error:
        return True
    return False

Crawlbase's documented billing rule also considers original_status. In this recorded matrix all 220 received responses had cb_status=200 and original_status=200, and the remaining call was the transport failure, so the two rules agree on this data. A body can fail this application's checks while still meeting the billing statuses. The unusable Explore Markdown responses did. The application's pass count and its billable-response count answer different questions.

Credits were not measured. No credit header was recorded, and the probe's account summaries came back with null usage fields. That is a reading error in the helper, not an empty account: the Account API nests the month's totals under currentMonth, and the helper reads totalSuccess and the other fields at the top level, so they are always null. Those summaries establish no measured usage delta:

python
def account(self):
    token = self._token("normal")
    try:
        response = self._session.get(
            "https://api.crawlbase.com/account",
            params={"token": token, "product": "crawling-api"},
            timeout=60,
        )
    except Exception as exc:
        return {"error": self._fetch.scrub(str(exc), self._secrets())}
    try:
        payload = response.json()
    except ValueError:
        return {"error": "account response was not JSON", "http_status": response.status_code}
    return {
        "http_status": response.status_code,
        "totalSuccess": payload.get("totalSuccess"),
        "totalFailed": payload.get("totalFailed"),
        "totalDue": payload.get("totalDue"),
        "remainingCredits": payload.get("remainingCredits"),
    }

All 220 received matrix responses recorded the complex tier. Under the documented credit multipliers, a Complex request uses two credits and JavaScript rendering doubles that. The matrix contained 100 received Normal-token responses and 120 received JavaScript-token responses:

Estimated matrix consumption: (100 × 2) + (120 × 4) = 680 credits.

That is an estimate from the recorded token mix, not an account reading, and it excludes the probe. The Google scraper product page advertises up to 5,000 free requests; check the actual credit balance and the applicable multipliers before you read an allowance as a number of Explore calls.

The runner is serial, uses the waits described above, and stops before another request once its billable cap is reached. If a lower cap interrupts a batch between a failed first attempt and its retry, the current runner can leave that partial batch out of the CSV; this did not happen in the recorded run. Within the cap, a challenge notice gets the same single retry as any other failed verdict, and the application adds no CAPTCHA-solving logic. Review Google's terms for the intended use and pick a schedule that matches the freshness your job needs. The Web Has Started Grading Crawlers covers the wider context for pacing and crawler behaviour.

Conclusion

The November 16 run showed why usable Google Trends data needs its own acceptance check. Every received matrix response reported successful HTTP and crawl statuses, yet Explore Markdown produced no usable timelines, some English Explore responses had empty charts, and localized pages exposed the limits of the markup parser.

Crawlbase's built-in scrapers supplied compact structured responses: Trending Now passed 20/20, and the JavaScript-token Explore scraper passed 18/20 after the geography correction. The application made those results inspectable by preserving statuses, checking the required fields, separating diagnostics from acceptance, and keeping each retry as its own attempt.

Start from the documented Google Trends scraper or Google Trends Explore scraper, apply checks suited to the data you asked for, and keep representative bodies whenever a verdict changes. That gives the pipeline a definition of success that goes beyond receiving a 200.

Crawlbase Crawling API

Fetch Google Trends as rendered HTML, Markdown, or structured JSON from the built-in Trending Now and Explore scrapers, with cb_status and original_status on every response so your own checks decide what counts. Start with up to 5,000 free requests, no card.

Frequently Asked Questions (FAQs)

They establish the reported acquisition statuses. This run still returned Oops pages, a reCAPTCHA notice, and empty timelines under those statuses. Require the fields and the coverage your application needs before you accept the data. A failed application verdict also does not make a received response non-billable.

Token choice depends on the surface and the path. Trending Now scraper JSON, Trending Now Markdown, and RSS passed with the Normal token in this matrix. The Explore scraper used the JavaScript token after the probe's Normal-token response came back with empty chart objects. Both rendered and no-wait HTML used JavaScript tokens. The harness retries the configured request without changing tokens on its own.

No. Explore values are normalized interest scores for the selected time and geography. Trending Now labels such as 1M+ are bucketed traffic for the selected trend period. Keep each field's meaning: neither an Explore score of 100 nor a 1M+ label is an exact count of searches that this application can recover.

Can the same checks be used for another country or a shorter time range?

Adapt them first. The CLI's --geo option changes the requested URLs, but the Explore scraper's acceptance rule still expects the United States. The 40-point minimum and the peak-of-100 check were chosen for a fixed single-term, twelve-month experiment, and localized markup may need parser changes. Test the intended geography, language, and time window with matching assertions before you compare its pass rate with this run.

Start Building

Crawl any site at scale, without fighting infrastructure.

Crawlbase handles proxies, fingerprints, and CAPTCHAs so your team ships data pipelines instead of maintaining crawl plumbing. Up to 5,000 requests free, no card required.

Self-serve · No sales call required · Enterprise crawl volumes available