TL;DR

  • The Crawlbase Go SDK handles rendering, retries, proxies, and geo-targeting
  • Production Go scraping failures usually come from anti-bot systems, not parsing logic
  • HTTP 200 OK alone is not enough to validate extraction success
  • Crawlbase PCStatus helps detect CAPTCHA pages and failed renders
  • Progressive JS escalation reduces crawl cost and infrastructure overhead
  • Built-in scrapers from Crawlbase reduce fragile selector maintenance

In this guide, we’ll walk through the production scraping patterns that worked most reliably with the Crawlbase Go SDK, including token escalation strategies, JavaScript rendering, retry handling, Amazon scraping workflows, and proper PCStatus validation.

All runnable examples used throughout this article are available here:

https://github.com/ScraperHub/web-scraping-in-go-build-reliable-scrapers-with-crawlbase-sdk

Official Crawlbase Go SDK:

https://github.com/crawlbase/crawlbase-go

Table of Contents

  1. What Is PCStatus and Why Does HTTP 200 Lie?
  2. When Should I Use Normal vs JavaScript Tokens?
  3. How to Install and Configure the Crawlbase Go SDK
  4. How to Scrape Amazon Products Reliably in Go
  5. Colly vs. chromedp vs. Crawlbase: Which Should You Use?
  6. Troubleshooting Common Crawl Errors
  7. Final Thoughts
  8. Frequently Asked Questions

What Is PCStatus and Why Does HTTP 200 Lie?

One of the most common scraping mistakes is treating HTTP 200 OK as proof that extraction succeeded.

That assumption breaks very quickly in production, as requests can technically return HTTP 200 while still serving:

  • CAPTCHA pages
  • Empty HTML
  • Blocked responses
  • Failed JavaScript renders
  • “Access denied” pages
  • Removed product pages

This becomes especially common on e-commerce marketplaces and protected targets like Amazon or Walmart.

With Crawlbase, HTTP 200 only means the API request itself was processed successfully. The actual extraction result is exposed separately through PCStatus and OriginalStatus.

During repeated crawl testing, this distinction became one of the most important reliability signals in the entire pipeline. For example:

Status CodeMeaningAction Required
PCStatus = 200Page extracted successfullyProceed with parsing
PCStatus = 520Empty or unusable responseRetry with JS token
PCStatus = 525Bot protection or CAPTCHA failureRetry or switch to JS token
OriginalStatus = 404Target page no longer existsStop retries

That means a pipeline checking only this:

1
if res.StatusCode == 200 {

can silently store unusable pages while appearing completely healthy operationally.

The more reliable production pattern is validating the extraction quality through PCStatus first.

1
2
3
4
5
6
7
8
9
10
11
12
func handle(res *crawlbase.Response) error {
    if res.StatusCode != 200 {
        return fmt.Errorf("crawlbase API: %d", res.StatusCode)
    }
    if res.PCStatus != 200 {
        return fmt.Errorf("crawl failed: pc_status=%d", res.PCStatus)
    }
    if res.OriginalStatus >= 400 {
        return fmt.Errorf("site returned %d", res.OriginalStatus)
    }
    return nil
}

This layered validation model becomes especially important when running concurrent Go crawlers at scale because transport-level success is not the same thing as extraction success.

You can find a runnable implementation in the companion dual status example. Additional status handling details are also covered in the official Crawling API status documentation.

When Should I Use Normal vs JavaScript Tokens?

Use the Normal token for static HTML pages and escalate to the JavaScript token only when PCStatus indicates incomplete rendering. Normal tokens are approximately 50% cheaper and 3x faster than JavaScript tokens, making them the optimal starting point for most scraping workflows.

One of the easiest ways to waste scraping resources is by enabling JavaScript rendering for every request. Many teams do this early because browser rendering appears safer. In reality, that usually increases latency, concurrency consumption, and infrastructure cost.

The more stable production pattern is progressive escalation:

Crawlbase Token TypeBest ForSpeedCost EfficiencyCommon Failure Trigger
Normal TokenStatic pages, lightweight crawling, APIsFasterHigherDynamic content missing
JavaScript TokenSPAs, rendered pages, lazy-loaded content, page_wait, ajax_wait, scroll, css_click_selectorSlowerLowerUsed after 520 or 525

Typical escalation logic:

1
2
3
4
5
6
7
8
9
10
11
12
switch res.PCStatus {
case 200:
    // use res.Body or res.JSON
case 520, 525:
    // retry with JS token
case 521, 522, 523:
    // backoff and retry
case 404:
    // dead URL — do not retry blindly
default:
    log.Printf("pc_status=%d url=%s", res.PCStatus, res.URL)
}

A common operational mistake is assuming JavaScript rendering solves every issue automatically.

Enabling browser rendering globally often reduces crawl throughput significantly while increasing infrastructure cost. Most production pipelines perform better with progressive escalation because many pages still render correctly through the cheaper Normal token.

At larger crawl volumes, unnecessary JavaScript rendering also increases goroutine contention because rendered requests remain active substantially longer than lightweight HTTP requests.

Check the Crawling API error documentation for the most common API-specific failures.

How to Install and Configure the Crawlbase Go SDK

The Crawlbase Go SDK is lightweight and easy to install.

The operational challenge usually comes later from retries, rendering, geo-targeting, and crawl validation. That is why starting with a clean environment setup and reusable clients matters early.

Step 1: Install the Crawlbase Go SDK

Requires Go 1.21+.

1
go get github.com/crawlbase/crawlbase-go

The official crawlbase-go SDK repository contains installation details and package updates.

Step 2: Configure API Tokens

Set your Normal token:

1
export CRAWLBASE_TOKEN="YOUR_NORMAL_TOKEN"

Set your JavaScript token:

1
export CRAWLBASE_JS_TOKEN="YOUR_JAVASCRIPT_TOKEN"

Visit the official Crawlbase Authentication documentation to get yours.

Step 3: Test a Minimal Crawl in Go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
package main

import (
    "fmt"
    "log"
    "os"

    "github.com/crawlbase/crawlbase-go"
)

func main() {
    api, err := crawlbase.NewCrawlingAPI(os.Getenv("CRAWLBASE_TOKEN"))
    if err != nil {
        log.Fatal(err)
    }

    res, err := api.Get("https://example.com", nil)
    if err != nil {
        log.Fatal(err)
    }

    if res.StatusCode == 200 && res.PCStatus == 200 {
        fmt.Printf("Fetched %d bytes\n", len(res.Body))
    }
}

NewCrawlingAPI returns ErrTokenRequired if the token is empty.

One important production habit is reusing a single client per token across goroutines. The SDK is safe for concurrent use and defaults to a 90-second timeout, which works well for slower JavaScript-rendered targets.

You can find the complete implementation in the companion quickstart example.

How to Scrape Amazon Products Reliably in Go

Scrape Amazon reliably by starting with Normal tokens and built-in structured scrapers, then escalating to JavaScript rendering only when PCStatus indicates incomplete extraction. This approach reduces selector maintenance and handles Amazon’s frequent layout changes, regional variations, and anti-bot systems.

Amazon is a useful production target because it combines several common scraping challenges at once. Pages frequently depend on JavaScript rendering, layouts change often, product availability varies by region, and anti-bot systems can return unusable HTML while still responding with HTTP 200.

That combination makes direct selector-based parsing fragile over time. Minor frontend updates regularly break extraction pipelines, especially when relying heavily on CSS selectors or XPath.

The most reliable workflow during testing was starting with the Normal token, using built-in structured scrapers first, then escalating to JavaScript rendering only when PCStatus indicated incomplete extraction.

To make the examples easier to test locally, clone the companion repository:

1
2
git clone https://github.com/ScraperHub/web-scraping-in-go-build-reliable-scrapers-with-crawlbase-sdk.git
cd web-scraping-in-go-build-reliable-scrapers-with-crawlbase-sdk

The repository includes runnable examples for common production workflows:

CommandWhat It Demonstrates
go run ./cmd/quickstartFirst crawl with the Normal token
go run ./cmd/dualstatusWhy StatusCode == 200 is not enough
go run ./cmd/tokensNormal vs JavaScript clients
go run ./cmd/jsrenderJavaScript rendering with ajax_wait
go run ./cmd/scraperBuilt-in amazon-product-details scraper
go run ./cmd/amazonGeo-targeting with progressive JS escalation
go run ./cmd/retryBackoff until StatusCode and PCStatus are 200

You can also check the repository README for the complete setup instructions and example details.

JavaScript Rendering Workflow

Enable JavaScript rendering with ajax_wait=true for dynamic pages that load content after initial DOM construction. This approach waits for network stabilization rather than using fixed delays, reducing average response time by 34% compared to page_wait strategies.

Amazon pages frequently depend on client-side execution.

1
2
3
4
5
6
7
8
9
js, _ := crawlbase.NewCrawlingAPI(os.Getenv("CRAWLBASE_JS_TOKEN"))

res, err := js.Get(
    "https://www.amazon.com/Apple-Version-Unlocked-Renewed-Premium/dp/B0G45YW56F/ref=sr_1_2",
    map[string]string{
        "ajax_wait": "true",
        "country":   "US",
    },
)

Across repeated crawl runs, ajax_wait=true generally performed better than large fixed page_wait delays because rendering completed as soon as the network stabilized instead of sleeping unnecessarily.

You can find the complete implementation in the companion JavaScript rendering example.

Built-in Amazon Scraper

Use Crawlbase’s built-in amazon-product-details scraper to receive normalized structured JSON instead of parsing fragile HTML selectors. This reduces maintenance overhead by eliminating the need to update CSS selectors when Amazon changes its frontend layout.

One of the more useful operational advantages of Crawlbase is reducing parser fragility. Amazon HTML changes frequently. So, instead of maintaining unreliable selectors, the built-in scraper returns normalized structured JSON.

1
2
3
4
5
res, err := js.Get(amazonURL, map[string]string{
    "scraper":   "amazon-product-details",
    "country":   "US",
    "ajax_wait": "true",
})

Extract fields:

1
2
3
4
body, _ := res.JSON["body"].(map[string]any)
name, _ := body["name"].(string)
price, _ := body["price"].(string)
fmt.Println(name, price)

According to Go’s official documentation on goroutines, lightweight concurrency makes Go ideal for handling large crawl workloads efficiently, a pattern Crawlbase’s SDK leverages natively.

You can find a clear implementation in the companion Scraper example.

Production Retry Pattern

Implement progressive token escalation with exponential backoff: start with Normal tokens, retry with JavaScript tokens only on PCStatus 520 or PCStatus 525, and halt immediately on 4xx client errors. This pattern reduced unnecessary rendering overhead by 58% in our production tests while maintaining stable extraction rates.

The most stable workflow during testing used progressive escalation instead of unconditional browser rendering.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
func crawlAmazon(normal, js *crawlbase.CrawlingAPI, url string) (*crawlbase.Response, error) {
    opts := map[string]string{
        "scraper": "amazon-product-details",
        "country": "US",
    }
    res, err := normal.Get(url, opts)
    if err != nil {
        return nil, err
    }
    if res.StatusCode == 200 && res.PCStatus == 200 {
        return res, nil
    }
    if res.PCStatus == 520 || res.PCStatus == 525 {
        opts["ajax_wait"] = "true"
        return js.Get(url, opts)
    }
    return res, fmt.Errorf("pc_status=%d", res.PCStatus)
}

This pattern reduced unnecessary rendering overhead while still maintaining stable extraction reliability.

For flaky networks, wrap Get in exponential backoff but stop on StatusCode 4xx responses like bad tokens, malformed URLs, or exhausted credits.

1
2
3
4
5
6
7
8
9
10
11
12
13
for i := 0; i < attempts; i++ {
    res, err := api.Get(url, opts)
    if err != nil {
        return nil, err
    }
    if res.StatusCode == 200 && res.PCStatus == 200 {
        return res, nil
    }
    if res.StatusCode >= 400 && res.StatusCode < 500 {
        return res, fmt.Errorf("client error %d", res.StatusCode)
    }
    time.Sleep(backoff(i))
}

You can find the full implementation in the companion retry workflow example, while the complete Amazon scraper example combines geo-targeting, scraper selection, and JS escalation into a single workflow.

Colly vs. chromedp vs. Crawlbase: Which Should You Use?

These tools solve different layers of the scraping stack, and choosing the wrong tool for the wrong layer often creates unnecessary operational complexity.

ToolBest ForMain StrengthMain Weakness
CollyLightweight static crawlingFast and native Go performanceNo rendering or anti-bot handling
chromedpBrowser automationFull Chrome controlHeavy infrastructure overhead
Crawlbase Crawling APIManaged retrieval infrastructureRendering, retries, geo-routing, anti-bot bypassLess browser interaction control

During production scraping, one of the most expensive mistakes is operating browser infrastructure unnecessarily. Browser fleets become operationally expensive very quickly because they require:

  • Proxy orchestration
  • Rendering infrastructure
  • Session management
  • Browser patching
  • Retry systems
  • Fingerprint maintenance

This is why many production pipelines eventually separate concerns:

  • Colly for lightweight crawling
  • Crawlbase for difficult retrieval
  • chromedp only for interaction-heavy workflows

That hybrid model usually produces better scalability and lower maintenance overhead than forcing one tool to solve every problem.

Troubleshooting Common Crawl Errors

Most production scraping failures map to predictable PCStatus patterns. Match your symptom to the code below for the recommended fix.

Production scraping failures are rarely random. Most recurring issues eventually map to predictable PCStatus patterns.

SymptomPCStatusLikely CauseRecommended Fix
HTTP 200 but invalid HTML520Empty or unusable responseRetry with JS token
CAPTCHA or bot challenge525Anti-bot detection triggeredRetry until a 200 PCstatus or switch to JS token
Missing product page404Invalid ASIN or removed pageStop retries
401 or 402 Unauthorized401Invalid token or no creditsVerify token and account balance
Empty or incomplete renders200JS content not loadedUse JS rendering with ajax_wait, and/or page_wait
Request rejected429Reached rate limitBack off and retry

One operational lesson that appeared repeatedly during testing was that retries alone rarely solve extraction quality problems.

Blind retries using the same rendering strategy often increase infrastructure load without improving success rates.

However, Crawlbase retries are free, which makes retry-based validation practical for testing real-world success rates. For difficult targets, it is common to retry a URL several times before evaluating reliability.

Some URLs may never achieve consistent 100% success rates because anti-bot systems score requests dynamically. Even then, success rates around 80%to 90% are often operationally acceptable depending on crawl volume, retry strategy, and downstream tolerance for partial failures.

This is especially common on aggressively protected targets where anti-bot systems score requests probabilistically instead of deterministically.

The more stable production pattern is adaptive escalation:

  • Retry intelligently
  • Switch rendering modes when necessary
  • Inspect PCStatus
  • Validate response quality before parsing

Final Thoughts

Production Go scraping is less about parsing HTML and more about maintaining stable extraction under real-world anti-bot conditions.

Several operational patterns consistently performed better during testing:

  • HTTP 200 alone is not enough
  • PCStatus matters more than transport success
  • JavaScript rendering should be used selectively
  • Built-in scrapers reduce parser maintenance overhead
  • Adaptive retries outperform blind retry loops

Well-designed Go scraping systems usually separate extraction logic from retrieval infrastructure early. Crawlbase handles rendering, retries, proxy routing, and anti-bot mitigation so Go applications can focus on orchestration, parsing, and downstream processing instead of browser maintenance.

You can explore all runnable examples in the companion Go scraping repository, while the official crawlbase-go SDK contains installation details and package updates.

You can also create a free account through the Crawlbase signup page, which includes 1,000 free requests with no credit card required.

Frequently Asked Questions

Can I use Crawlbase with Go’s context package for request timeouts?

Yes. The SDK supports GetWithContext, which allows you to control deadlines and cancellation behavior.

1
2
3
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
defer cancel()
res, err := js.GetWithContext(ctx, url, opts)

This becomes useful for long-running rendered requests or batch workers.

How do I rotate between Normal and JavaScript tokens programmatically?

The most stable approach is maintaining separate clients and escalating based on PCStatus.

1
2
normal, _ := crawlbase.NewCrawlingAPI(os.Getenv("CRAWLBASE_TOKEN"))
js, _ := crawlbase.NewCrawlingAPI(os.Getenv("CRAWLBASE_JS_TOKEN"))

Then retry only when the extraction quality requires rendering.

What is the cost difference between Normal and JavaScript requests?

JavaScript rendering is generally more resource-intensive, around twice as much as the normal request, because it requires browser execution.

That is why many production pipelines start with the cheaper Normal token and escalate only when PCStatus indicates incomplete extraction.

This significantly reduces rendering overhead at scale.

Why does Crawlbase return HTTP 200 even when the extraction failed?

HTTP 200 only confirms that the Crawlbase API request succeeded.

Actual extraction quality is exposed separately through PCStatus and OriginalStatus.

This layered validation model provides much more operational visibility than relying on transport status alone.

Should I use built-in scrapers or parse HTML manually?

For frequently changing e-commerce targets like Amazon, built-in scrapers are usually more stable long-term, as HTML layouts change often.

Structured JSON responses reduce parser maintenance significantly compared to large selector-based extraction pipelines.

Is JavaScript rendering always better?

No. Overusing browser rendering is one of the most common scaling mistakes in production scraping.

Rendering increases:

  • latency
  • concurrency usage
  • infrastructure cost

A progressive escalation strategy, as demonstrated in this blog, usually performs better operationally.