Monitor Web Scraping with Prometheus and GrafanaSeparate the Crawl Outcome, the Target's Answer, and the Bill

A missing page and a blocked crawl need different responses. In our live run, a request for a page that does not exist on example.com came back as HTTP 200 with cb_status=200 and original_status=404. The crawl succeeded and was billable. A Normal-token request to Glassdoor came back as HTTP 520 with cb_status=705 and original_status=403. That crawl failed and was free. A monitor that reads only the outer HTTP status cannot tell those two apart.

This post turns those signals into a web scraping monitoring dashboard. A small client records each finished attempt, Prometheus collects counters and histograms, and Grafana shows per-domain health next to account-wide usage from the Account API. Three alert rules cover sustained acquisition failures, Normal-token responses that need a routing decision, and a budget that is projected to run out.

The companion repository contains the complete stack, nine walkthrough folders, unit tests, and live verification scripts. The figures below come from the November 2026 live run, which deliberately mixed easy targets, a protected site on the wrong token, and a URL that returns HTTP 500. They show what the monitoring does with failures. They are not a Crawlbase success-rate benchmark.

Separate crawl health from the target site's response

The Crawling API returns metadata that answers three different questions: did Crawlbase process the request, what happened during acquisition, and what did the target site answer?

In the recorded HTML responses the tier arrived as the x-domain-complexity header. With format=json it appeared as domain_complexity in the envelope. The signal parser accepts both spellings, normalises header names, and reads the JSON envelope when the request asked for it. It also accepts the legacy pc_status alias, while the application exposes cb_status everywhere.

The raw-request walkthrough prints those signals for a few URLs. These two observations come from the run:

https://example.com/does-not-exist
  http_status       200
  cb_status         200
  original_status   404
  domain_complexity standard
  seconds           1.4

https://www.glassdoor.com/Reviews/index.htm
  http_status       520
  cb_status         705
  original_status   403
  domain_complexity standard
  seconds           20.3

The diagram above shows a different Glassdoor request from the same run, which failed with cb_status=576; the listing is the one that returned 705. Both codes belong to the same failure family. The missing page completed in 1.4 seconds. Glassdoor returned its failed response after 20.3 seconds, and a second request to the same URL took 68 seconds before failing. A 404 can therefore be a successful acquisition even though it contains nothing to extract. Required-field validation belongs after acquisition and sits outside this dashboard's success definition.

The run also returned complex_ii as a tier, so the parser accepts any short lowercase slug instead of a fixed list of three names. That matches the wider tier range on the pricing page. Observed cb_status values included 0, 515, 521, 576, 611, and 702 alongside the documented outcomes. The status reference explains the common codes and their families. Keep unfamiliar codes visible on the dashboard; a code number on its own does not establish a root cause.

HTTP 520 identifies a failed crawl, but it is not the only error response. Authentication, rate limiting, and request validation produce other HTTP statuses, and a transport timeout produces no HTTP response at all.

Record each attempt without creating a series for every URL

The instrumented client makes one Crawling API request per get() call. It does not raise on HTTP 520; a failed crawl is a normal, labelled outcome. Transport errors are counted and then re-raised, so retry policy stays with the caller. The client never switches tokens or retries a URL on its own. This is the response path as it appears in the repository:

def get(self, url: str, **params: Any) -> CrawlResult:
    domain = self._limit_domain(domain_label(url))
    token_type = "js" if str(params.get("javascript", "")).lower() == "true" else self.token_type
    success_codes = self.custom_success_codes | parse_success_codes(params.get("custom_success_codes"))

    self._ensure_series(domain, token_type)
    started = self._clock()
    try:
        response = self._http.get(
            f"{self.base_url}/", params={"token": self._token, "url": url, **params}
        )
    except httpx.HTTPError as exc:
        self._record_failure(domain, token_type, exc)
        raise
    elapsed = self._clock() - started

    json_body = None
    if params.get("format") == "json":
        try:
            json_body = response.json()
        except ValueError:
            json_body = None
    signals = parse_signals(response.status_code, response.headers, json_body)
    billable = is_billable(signals.cb_status, signals.original_status, success_codes)

    # Counted on completion, next to the outcome, so success <= attempts in any window.
    self.metrics.attempts.labels(domain=domain, token_type=token_type).inc()
    self.metrics.requests.labels(
        domain=domain,
        cb_status=status_label(signals.cb_status),
        original_status=status_label(signals.original_status),
        http_status=status_label(signals.http_status),
        token_type=token_type,
        billable=str(billable).lower(),
    ).inc()
    self.metrics.latency.labels(domain=domain, complexity=signals.complexity).observe(elapsed)
    self.metrics.response_bytes.labels(domain=domain).observe(len(response.content))

    return CrawlResult(url, domain, token_type, signals, billable, elapsed, response.content)

The metric definitions separate finished attempts from individual outcomes. crawlbase_attempts_total increments on every received response and every transport error. crawlbase_requests_total carries the status labels; a transport error is recorded with none for all three statuses and billable="false", so timeouts stay in the success-rate denominator:

def _record_failure(self, domain: str, token_type: str, exc: httpx.HTTPError) -> None:
    reason = "timeout" if isinstance(exc, httpx.TimeoutException) else "transport"
    self.metrics.attempts.labels(domain=domain, token_type=token_type).inc()
    self.metrics.client_errors.labels(domain=domain, token_type=token_type, reason=reason).inc()
    # Count it as a request too, so success-rate denominators include it.
    self.metrics.requests.labels(
        domain=domain, cb_status="none", original_status="none",
        http_status="none", token_type=token_type, billable="false",
    ).inc()

Labels decide how many series Prometheus has to store. The domain helper lowercases and IDNA-encodes the hostname, strips a leading www., and drops the path and query. Subdomains stay distinct. The demo shares one DomainLimiter across both token clients; it keeps up to 200 domain names and files everything after that under other.

Status labels accept integers from 0 to 999, with none for a missing value and other for anything out of range. The complexity label accepts a lowercase slug of at most 24 characters. Those caps bound label growth, although the combinations of status, domain, and tier still multiply. Full URLs and request IDs belong in diagnostic logs, not in metric labels. The demo also installs a logging filter that masks token= query parameters, the configured token values, and /crawler/<token> paths in every log line.

Establish counter baselines before traffic starts

A counter that Prometheus first sees at 1 has no observed transition from zero. rate() cannot recover that first event from later samples. The demo therefore preregisters its known domains and waits 20 seconds before the first request, so the 15-second scrape sees the zero values:

def preregister(self, urls: Iterable[str], token_type: str | None = None) -> None:
    """Create zero-valued series before traffic starts.

    Prometheus only sees a counter once it exists. A series born at 1 between
    two scrapes shows no increase, so the first failure would be invisible.
    """
    for url in urls:
        self._ensure_series(self._limit_domain(domain_label(url)), token_type or self.token_type)

def _ensure_series(self, domain: str, token_type: str) -> None:
    if (domain, token_type) in self._known:
        return
    self._known.add((domain, token_type))
    self.metrics.attempts.labels(domain=domain, token_type=token_type)
    self.metrics.requests.labels(
        domain=domain, cb_status="200", original_status="200",
        http_status="200", token_type=token_type, billable="true",
    )

preregister() creates the attempts series and the default 200/200 success series for each domain and token type. It does not create every possible outcome. A status combination that appears for the first time can still lose its first event from a rate-based breakdown. The separate attempts baseline makes the success denominator reliable without requiring every failure label in advance.

Classify billable responses separately from successful extraction

The billing classifier follows the Crawling API's documented rule:

CHARGED_CB_STATUSES = frozenset({200, 207})
CHARGED_ORIGINAL_STATUSES = frozenset({200, 201, 301, 302, 303, 307, 308, 404, 410})


def is_billable(
    cb_status: int | None,
    original_status: int | None,
    custom_success_codes: Iterable[int] = (),
) -> bool:
    if cb_status not in CHARGED_CB_STATUSES or original_status is None:
        return False
    return original_status in CHARGED_ORIGINAL_STATUSES or original_status in set(custom_success_codes)

A response needs both an eligible cb_status and an eligible target status to be charged. A successful crawl of a 404 or 410 page is billable. If you pass custom_success_codes=403, Crawlbase treats a 403 answer as a successful crawl, returns cb_status=200, and charges for it. The local classifier mirrors that rule: it accepts a custom code only together with a 200 or 207 cb_status, so it never upgrades a response that Crawlbase itself marked as failed.

The billing walkthrough includes a decision table and 41 offline tests. The metrics-endpoint walkthrough exposes the counters through start_http_server(8000). Two recorded lines show the labels in use:

crawlbase_requests_total{billable="true",cb_status="200",domain="example.com",http_status="200",original_status="404",token_type="normal"} 9.0
crawlbase_requests_total{billable="false",cb_status="576",domain="glassdoor.com",http_status="520",original_status="403",token_type="normal"} 3.0

The first series records nine billable acquisitions of a missing page. The second records three free Glassdoor failures. Neither says whether an extracted record was valid.

The dashboard defines acquisition success as cb_status=200. The billable category also includes qualifying 207 responses. A 207 is an empty response, so it can be billable while falling outside the success policy. Keep those definitions separate when you read success rate and billable share side by side.

Compare local activity with account-wide usage

Local counters describe one instrumented process. The Account API returns monthly account-level statistics for one product, including successes, failures, the amount due, and per-domain summaries. It allows one request per five minutes, which makes it a different kind of source from a /metrics endpoint scraped every 15 seconds.

The account poller keeps the last snapshot and the time of the last call on disk. It writes the timestamp before it sends the request, so a failed call or an interrupted process still consumes the 300-second interval:

def poll(self, metrics: AccountMetrics | None = None) -> AccountSnapshot | None:
    """Return a fresh snapshot if allowed, otherwise the cached one."""
    if self.seconds_until_allowed() > 0:
        if metrics:
            metrics.cache_hits.inc()
        return self._cached

    self._last_call = self._clock()
    self._save_state()
    try:
        response = self._http.get(
            f"{self.base_url}/account", params={"token": self._token, "product": self.product}
        )
        response.raise_for_status()
        snapshot = parse_account(response.json())
    except (httpx.HTTPError, ValueError) as exc:
        result = "timeout" if isinstance(exc, httpx.TimeoutException) else "error"
        status = exc.response.status_code if isinstance(exc, httpx.HTTPStatusError) else None
        log.warning("Account API call failed (%s): %s status=%s", result, type(exc).__name__, status)
        if metrics:
            metrics.api_calls.labels(result=result).inc()
        return self._cached

    self._cached = snapshot
    self._save_state()
    if metrics:
        metrics.api_calls.labels(result="ok").inc()
    return snapshot

On startup the poller reloads the saved call time and snapshot. The reconciler walkthrough shows a second call answered from cache immediately, and in the recorded test a new process also reused the persisted response.

Run one reconciler per token. The state file supports sequential restarts, not coordination between concurrent pollers. Each instance keeps its own in-memory timestamp and takes no interprocess lock. During testing, two pollers with separate state files produced an early-call rejection, and a shared file alone does not make concurrent pollers safe.

The documented field table describes totals and domain breakdowns. The recorded payload nested them under currentMonth and metrics:

{
  "currentMonth": {
    "totalSuccess": "<str>",
    "totalFailed": "<str>",
    "totalDue": "<float>",
    "domainStats": [
      {
        "domain": "<str>",
        "metrics": {
          "totalRequests": "<int>",
          "success": "<int>",
          "failure": "<int>",
          "successRate": "<str>"
        }
      }
    ]
  }
}

The parser accepts this observed structure and a flat one. It converts string totals and percentage strings such as "100.0%", and where domain counts are present it computes the ratio from successes and total requests. The live domain names were bare labels such as glassdoor rather than glassdoor.com. The data lagged traffic by several minutes, and product=crawler returned the same monthly structure.

Prometheus can scrape the reconciler's gauges every 60 seconds without triggering an Account API call on each scrape. Calls that land inside the guard interval return cached data, and the long-running reconciler simply waits until the next call is allowed.

The recording rules derive a site label so hostnames can be compared with the account's bare labels. It is a plain hostname regex, not public-suffix-aware resolution. It works for the .com and .org targets in the demo; adapt the mapping before pointing it at example.co.uk.

Understand the budget fallback

remainingCredits was absent from the recorded Crawling API response. The documentation notes that it is only returned for subscription-based products, so the reconciler needs a fallback. With CRAWLBASE_MONTHLY_BUDGET set, the snapshot is exported as max(0, budget - totalSuccess) under source="budget". When the API does supply remaining credits, the gauge uses source="account_api" instead:

metrics.total_success.set(snapshot.total_success)
metrics.total_failed.set(snapshot.total_failed)
metrics.total_due.set(snapshot.total_due)
if snapshot.remaining_credits is not None:
    metrics.remaining_credits.labels(source="account_api").set(snapshot.remaining_credits)
elif monthly_budget:
    metrics.remaining_credits.labels(source="budget").set(
        max(0.0, monthly_budget - snapshot.total_success)
    )

The fallback is a request-count estimate. The days-to-zero rule divides the remaining gauge by the last hour's billable-request rate multiplied by 86,400, which assumes one credit per billable request. JavaScript rendering and higher complexity tiers consume more credits, and other products can draw on the same balance. The forecast is good enough to demonstrate budget alerting. Use credit-weighted consumption before you treat it as a financial limit.

Connect the exporters, Prometheus, and Grafana

Acquisition and reconciliation run as separate processes. Each exposes a metrics endpoint, Prometheus scrapes both and evaluates recording rules, and Grafana queries those rules for panels and alerts.

The Compose walkthrough introduces four services: the scraper on port 8000, the reconciler on 8001, Prometheus on 9090, and Grafana on 3000. The final stack pins Prometheus 3.1.0 and Grafana 11.4.0, and its Python requirements pin HTTPX 0.28.1, prometheus-client 0.21.1, and python-dotenv 1.0.1. The scrape configuration is short:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: crawlbase-scraper
    static_configs:
      - targets: ["scraper:8000"]

  # The gauges only change every 5 minutes, a slower scrape is plenty.
  - job_name: crawlbase-account
    scrape_interval: 60s
    static_configs:
      - targets: ["reconciler:8001"]

Rules are evaluated globally every 15 seconds, with the Crawlbase recording group set to 30 seconds. Grafana refreshes the dashboard every 30 seconds and evaluates its alert group every minute. None of those schedules is the five-minute Account API guard; they are separate clocks.

The named cbwatch-account-state volume is shared by the walkthrough stacks and the final stack. A plain docker compose down keeps it, so you can move between stacks without discarding the guard. Removing the volume removes the saved call time with it.

The dashboard walkthrough introduces recording rules so that panels and alerts share one definition. The central per-domain ratio is:

# Success = the crawl worked (cb_status 200). The denominator is
# crawlbase_attempts_total, which also counts timeouts and is created at 0.
- record: crawlbase:domain_success_ratio:rate5m
  expr: |
    clamp_max(
      sum by (domain) (rate(crawlbase_requests_total{cb_status="200"}[5m]))
      /
      sum by (domain) (rate(crawlbase_attempts_total[5m])),
      1
    )

The numerator counts cb_status=200. The denominator is every finished attempt, timeouts included, with both token types combined per domain. clamp_max caps the ratio at 1 when two separately sampled counters produce a transient overshoot. The one-hour summary uses the same definition over a longer window.

The Prometheus rule tests feed synthetic counter sequences through the rules. Their Walmart sequence yields 90% success, and a second sequence confirms that timeouts lower the ratio. Those are fixtures, not the live observations.

Read each dashboard panel against its own definition

Grafana loads the dashboard from provisioned JSON. The dashboard generator builds that JSON from Python, and a unit test checks that the committed file matches the generator. The success-rate panel points at the shared rule:

timeseries("Success rate per domain", [target("crawlbase:domain_success_ratio:rate5m", "{{domain}}")],
           0, 6, unit="percentunit", minv=0, maxv=1,
           desc="cb_status 200 share per domain, 5 minute window. Alert fires under 90% for 10m."),

The schematic arranges the recorded observations by operational question. Read every value with its query window; a one-hour rate, a five-minute rate, and a raw counter total answer different questions and should not be combined into a single batch success fraction.

The status breakdown held 46 responses at 200, six client timeouts, four at 576, four at 0, two at 704, two at 705, and one each at 521 and 611. All 46 responses at 200 were billable and the other 20 were free, so the billable label and the status breakdown describe the same 66 attempts. The 20 free outcomes were all on glassdoor.com and httpbin.org. Every non-200 cb_status arrived as HTTP 520; the timeouts had no response and were recorded as none. Windowed rates and raw counters still differ: the one-hour success rate is not the raw 46-of-66 ratio.

The 87.8-second p95 needs a precise denominator. The client observes latency only once a response arrives, HTTP 520 included. Client timeouts never enter the histogram. The value is a bucket interpolation, not an exact percentile over individual durations, and the top bucket boundary is the 90-second timeout. Ordinary pages were observed around two seconds. HTTPX's 90-second timeout applies to each network operation, including waiting for the next chunk, so it is not a strict wall-clock deadline, as the HTTPX timeout documentation explains.

The gap between local and account figures has several expected causes. Local metrics cover this process's attempts and include client-side errors. The Account API covers the whole month for every caller using the selected product and can lag recent traffic. Those scopes do not give you a one-to-one invoice reconciliation or identical timeout accounting.

The data-age panel means less than its label suggests. The reconciler updates its timestamp whenever it applies a snapshot, including cached data after a failed poll or a restart. Read it next to the API error counters; on its own it does not prove the upstream data was freshly fetched.

Make scraping success rate alerting actionable

The alert walkthrough provisions three Grafana-managed rules. They separate a sustained acquisition loss from a token-routing investigation and a budget warning:

The rate gate on the domain alert is roughly one request every 125 seconds, or 2.4 requests per five-minute window. It filters nearly idle domains, but it is not a large-sample guarantee. Set the traffic floor and the threshold for the volume and reliability your job actually needs.

- uid: cb-domain-success-low
  title: Crawlbase domain success rate below 90%
  condition: C
  for: 10m
  noDataState: OK
  execErrState: Error
  labels:
    severity: page
  annotations:
    summary: "Success rate for {{ $labels.domain }} is {{ humanizePercentage $values.B.Value }}"
    description: >-
      cb_status 200 share has been under 90% for 10 minutes. Open the cb_status
      breakdown panel: 5xx/6xx codes are the crawl layer, 704/705/706 mean the
      target site said no (check original_status).
    runbook_url: https://crawlbase.com/docs/status-codes/
  data:
    - refId: A
      relativeTimeRange: { from: 600, to: 0 }
      datasourceUid: prometheus
      model:
        refId: A
        instant: true
        # Ignore domains with almost no traffic (under ~1 request per 2 minutes).
        expr: crawlbase:domain_success_ratio:rate5m and on (domain) crawlbase:domain_requests:rate5m > 0.008
    - refId: B
      datasourceUid: __expr__
      model: { refId: B, type: reduce, expression: A, reducer: last }
    - refId: C
      datasourceUid: __expr__
      model:
        refId: C
        type: threshold
        expression: B
        conditions:
          - evaluator: { type: lt, params: [0.9] }

The blocked rule matches 503|207|525|576|705 on the Normal token. The documentation suggests the JavaScript token for 207 and 503; the project keeps 525 for older responses and adds the observed 576 and 705 outcomes. Those additions are an application policy, not a product rule. A persistent 403 still calls for a look at the target's response and your request rate rather than an assumption that a token switch fixes every case.

In the live run, the Normal-token alert fired first, for glassdoor.com. The domain-success alert fired three minutes later, and the budget alert went pending once its rate estimate settled.

The repository provisions rules and severity labels, not Slack, email, PagerDuty, or a ticketing integration. Configure Grafana contact points and notification policies to route severity=page and severity=ticket where they belong. A label on its own does not page anyone.

An isolated 520, one slow response, or a known 404 belongs on the dashboard, not in a page. A sustained change in failure rate, completion time, or required-field coverage deserves a person. The provisioned rules treat missing data as OK, so monitor exporter availability and stopped jobs separately.

Grafana annotations use a single $ in references such as $labels.domain and $values.B.Value. The recorded Grafana 11 behaviour left doubled dollar signs literal, so the repository tests the template spelling along with the thresholds.

Extend the same metrics to Crawler queues and token routing

For the Crawler, the optional stats poller reads /crawler/<TOKEN>/stats and exports account totals plus per-crawler waiting, concurrency, latency, and paused gauges. With CRAWLER_STATS_ENABLED=true the reconciler attempts those polls on a nominal 60-second schedule, which synchronous API calls can delay.

The Crawler and multi-token walkthrough also keeps one client per token and routes Glassdoor through a configured JS_DOMAINS set. In the recorded run, the JavaScript-token request returned a clean 200. The operator updates that set after investigating an alert; the alert never rewrites routing on its own:

# Filled from the alert: Normal token gets cb_status 503/576/705 on these.
JS_DOMAINS = {"glassdoor.com"}

URLS = [
    "https://example.com/",
    "https://www.glassdoor.com/Reviews/index.htm",
]


def main() -> None:
    load_dotenv(find_dotenv(usecwd=True))
    clients = {
        "normal": InstrumentedCrawlingAPI(os.environ["CRAWLBASE_NORMAL_TOKEN"], "normal"),
        "js": InstrumentedCrawlingAPI(os.environ["CRAWLBASE_JS_TOKEN"], "js"),
    }
    for url in URLS:
        token_type = "js" if domain_label(url) in JS_DOMAINS else "normal"
        r = clients[token_type].get(url)
        print(f"{token_type:<6} http={r.status_code} cb_status={r.headers.get('cb_status')} {url}")

Both clients write to the same metric families with different token_type values, so you can inspect token-specific failures while keeping one per-domain view. Crawler names should identify stable queues in the same way, never individual jobs or URLs.

Run the stack and verify the integration

From the repository root, start the final stack with:

cd final
cp .env.example .env          # add CRAWLBASE_NORMAL_TOKEN and CRAWLBASE_JS_TOKEN
docker compose up -d --build
# Grafana: http://localhost:3000/d/crawlbase-observability (admin / admin)

The final environment template includes the budget and the optional Crawler settings. The root template supplies tokens for the walkthrough folders, and generated .env files are ignored by git. The final stack sends 40 requests by default and then keeps serving metrics. Successful acquisitions consume credits. Set a Grafana password before exposing the stack beyond your machine.

The README describes each stage and check. In the published layout, final/ and steps/ sit directly under the root; README commands that keep a code/ prefix refer to the earlier parent layout.

The project ships 319 offline tests using pytest and respx, plus six live tests selected separately with a marker. Offline verification passed all 319 and the walkthrough's 41 billing tests. The mocked HTTP tests cover the parsers, status classification, metric updates, cache behaviour, and restart handling.

The configuration checker runs Prometheus config and rule checks, the synthetic rule tests, and dashboard validation. The smoke test checks a running stack's targets, exported series, recording rules, dashboard, datasource, and alert-rule evaluation. The live probe, sample capture, and panel-query script expose the response shapes and queries behind the figures in this post.

Conclusion

Useful scraping observability starts by keeping three things apart: the crawl outcome, the target's answer, and your own acceptance policy. Crawlbase supplies the acquisition signals, the wrapper records them, Prometheus turns them into windowed metrics, and Grafana makes sustained change visible.

The live run showed why the distinctions matter. Valid 404 acquisitions were billable, a protected target produced several failure codes, client timeouts needed their own accounting, and account-wide statistics differed from the local one-hour view. The project makes all of that inspectable without labelling every page or embedding a monitoring platform inside the scraper.

Use the complete implementation with a small set of representative targets, confirm your success definition and credit assumptions, and connect the alerts to a real on-call response. Then measure the targets your application actually depends on.

Frequently Asked Questions (FAQs)

What should count as a successful scraping request?

This project counts cb_status=200 as acquisition success and puts every finished attempt in the denominator. A target 404 meets that definition; an empty 207 does not. For a product or article pipeline, add separate metrics for records that pass your required-field and content checks. Successful acquisition and usable business data are different questions.

Why does the Account API success rate differ from Grafana's local rate?

The local summary is a rolling one-hour window over this process's attempts. The Account API reports month-to-date activity for the selected product, can include other callers, and lagged traffic by several minutes in the recorded run. Align the scopes before reading a gap as missing traffic or incorrect billing.

Does the dashboard show the exact number of credits remaining?

When the Account API returns remainingCredits, which it does only for subscription-based products, the reconciler exports that balance. Otherwise it subtracts the month's successful requests from a configured budget. That fallback and the days-to-zero rule assume one credit per request. Account for rendering, domain multipliers, and shared product usage before relying on the estimate as a real balance.

Does the blocked-token alert retry with the JavaScript token automatically?

No. The client records the result and leaves retries to the caller. The multi-token example routes domains through a configured set, and the alert tells an operator where to look. Updating that routing and wiring notification destinations are explicit application and operations decisions.

NZ

Neil Zamora

Senior Architect · Crawlbase

Senior architect at Crawlbase, focused on the systems behind large-scale crawling: proxy rotation, anti-bot resilience, and the APIs that hide that complexity.

IK

Ian Kalvin

Technical Support Engineer · Crawlbase

Technical support engineer at Crawlbase, writing from the front line of what actually breaks in production scraping and proxy setups.