Build a Web Scraping Pipeline with ZapierDispatch the Crawl, Catch the Callback, Stop Waiting on Slow Pages

A Zap that scrapes a page usually works on the first try and starts failing later. The pattern is familiar: a Webhooks by Zapier action calls a scraping endpoint, waits for the HTML, and hands it to a Google Sheets row. It passes in the editor against a fast test page. Then it meets a real target that renders behind JavaScript, sits behind a challenge, or simply answers slowly, and the action runs out of time.

The instinct is to blame the request. The request is fine. The problem is that one Zapier action is holding open two operations with very different lifetimes: a workflow step measured in seconds, and a page retrieval that can take much longer whenever the target decides it should.

The fix is to stop making one wait for the other. Zapier dispatches the crawl and finishes. Crawlbase runs the crawl and reports back when it has something. This guide builds that pipeline with two Zaps, no code, and a recovery path for the runs that go wrong.

Why synchronous scraping breaks down

A Zapier action has a finite execution window. That is a reasonable design for an automation platform: steps are expected to be short, and a platform running millions of them cannot let any single step hold a slot indefinitely.

Web retrieval does not respect that expectation. The time to fetch a page is set by the target, not by you. A product page might answer in 400 milliseconds at 2am and take twenty seconds during a sale. A JavaScript-heavy listing has to be rendered before there is anything worth returning. A site under load, or one that decides to serve a challenge, can stretch a fetch far past anything a workflow step budgeted for.

The result is a workflow that is not broken so much as unpredictable. It succeeds on quick pages and times out on slow ones, which means it fails precisely on the targets that were worth automating.

Retrying does not help, because a retry repeats the same coupling with the same target. Neither does splitting the URL list into smaller batches, since the failure is per request, not per volume. What has to change is the shape of the workflow: from request, wait, process to dispatch, crawl, callback, process.

The asynchronous architecture

The Crawlbase Crawling API supports that separation directly. In its default synchronous mode the HTTP request stays open until the page is ready and the body comes back on that same response. With async=true the API instead accepts the job, returns a request ID (rid) straight away, and runs the crawl in the background. Adding callback=<webhook URL> tells it where to POST the finished page.

The distinction that matters here is between request accepted and crawl completed. The Dispatcher only ever learns the first. The page itself arrives at a different Zap, at a time nobody scheduled, which is exactly why the pipeline stops being sensitive to how slow the target is.

Two details are easy to get wrong. The parameter is callback, not callback_url. And for anything you intend to run in production, pair it with store=true, which saves the response in Crawlbase Cloud Storage against the same rid. That single flag is the difference between a lost callback costing you a lookup and costing you another crawl.

If your targets are fast and straightforward, you may not need any of this. The native Crawlbase Zapier integration gives you Crawl URL, Scrape Structured Data, and Take Screenshot as ordinary Zap actions with no webhook plumbing at all. Reach for the callback pattern when retrieval time is the thing breaking your workflow.

Build the receiver before the dispatcher

The two Zaps have a build-order dependency that catches people out. The Receiver's Catch Hook generates the URL that the Dispatcher must pass as callback, so the Receiver has to exist first. Building the Dispatcher first leaves you with a crawl request and nowhere to send the result.

Before starting, you need:

1. A Crawlbase account. Sign up and copy your Crawling API token from the dashboard. The Normal token covers most pages; the JavaScript token is for targets that need rendering.
2. A Zapier account on a plan that allows multi-step Zaps, since both Zaps here are multi-step.
3. A destination for the data: a Google Sheet, a Slack channel, an Airtable base, or a CRM.
4. A target URL to test with. Use a real page from the workflow you are automating rather than a placeholder, so the timing you see is the timing you will get.

Zap B: the receiver

The Receiver is the callback endpoint. It never starts a crawl. It sits waiting for Crawlbase to POST a finished page, then validates and routes it.

Step 1: create the Catch Hook

In Zapier, create a new Zap and choose Webhooks by Zapier as the trigger, with the event Catch Hook. Zapier generates a Custom Webhook URL that looks like this:

https://hooks.zapier.com/hooks/catch/1234567/abcdef/

Copy it. This is the value of the Crawlbase callback parameter in the Dispatcher, and it is the only thing connecting the two halves of the pipeline.

Step 2: give it a sample payload

Zapier needs to see one request before it can offer the callback fields for mapping. You can use its own test tools, or POST a sample yourself. A real Crawlbase callback carries the crawled content along with metadata including rid, url, original_status, and cb_status.

Mapping against a real callback rather than an invented sample is worth the extra minute. Field names in a hand-written test payload have a way of not matching what actually arrives.

Step 3: add the processing actions

With fields available, add whatever should validate, transform, and store the result. Formatter by Zapier handles trimming, substring extraction, and splitting without code. A Google Sheets: Create Spreadsheet Row action might map:

Google Sheets is the easiest thing to verify against, but the destination is interchangeable: Slack, Salesforce, Airtable, Notion, or anything else Zapier connects to. None of these actions run while the Dispatcher is open, because the Dispatcher finished long before the callback arrived.

Step 4: turn the receiver on

Publish the Receiver before you send a single real async request. A Catch Hook belonging to a Zap that is switched off will not process what it is sent, and a callback that arrives at a disabled endpoint is a crawl you have paid for and thrown away. Turning the Receiver on first is the cheapest habit in this whole build.

Zap A: the dispatcher

The Dispatcher takes a URL from an existing workflow, submits it for asynchronous crawling, records the rid, and ends. That is the entire job.

Step 1: choose the trigger

The trigger is whatever produces URLs in your business:

• Schedule by Zapier for recurring price or availability checks
• Google Sheets: New Spreadsheet Row to crawl a URL as it is added to a sheet
• Google Forms: New Response to crawl a URL someone submits

Map the field holding the fully qualified URL, including the scheme.

Step 2: add the Crawlbase request

Add Webhooks by Zapier as the action, with the event GET, and set the request URL to https://api.crawlbase.com/. GET matches how the Crawling API documents its parameters. POST works too, as long as the parameters reach the endpoint intact.

Step 3: configure the parameters

Under Query String Params, add five entries:

Two things to watch. The url value must be URL-encoded, which matters most when the target URL carries its own query string. And the token belongs in the Zap configuration and nowhere else: keep it out of screenshots, shared Zap templates, and support threads, and rotate it if it does get out.

For targets that need rendering, use the JavaScript token and add the rendering parameters, such as page_wait, documented with the Crawling API. Adding format=json is also worth considering: it returns status, URL, and body as one JSON envelope, which usually makes field mapping in the Receiver simpler.

The equivalent request, if you want to confirm the setup outside Zapier first:

curl -G 'https://api.crawlbase.com/' \
  --data-urlencode 'token=YOUR_CRAWLBASE_TOKEN' \
  --data-urlencode 'url=https://example.com/product/123' \
  --data-urlencode 'async=true' \
  --data-urlencode 'callback=https://hooks.zapier.com/hooks/catch/1234567/abcdef/' \
  --data-urlencode 'store=true'

Running that by hand is a good way to separate a Crawlbase problem from a Zapier problem before both are wired together.

Step 4: record the request ID

Add one more action that writes the rid somewhere durable, along with the input URL and the dispatch timestamp. A spreadsheet row is enough.

This is not bookkeeping for its own sake. The rid is the correlation key across all three places the crawl exists: the dispatch, the eventual callback, and the stored copy. It is what lets you answer whether a URL was already crawled, whether a callback was already processed, and where a missing result went.

Step 5: turn the dispatcher on

Publish it, and resist adding one more step. Any action that tries to read the page body from the Dispatcher's response reintroduces the wait you just removed, and the timeout comes back with it. The Dispatcher's response contains an rid. The page belongs to the Receiver.

Reading the callback: cb_status and original_status

When the crawl finishes, Crawlbase POSTs the result to the Catch Hook. Use Test trigger after a real crawl, or open a recent Zap run, and map from what actually arrived. The fields that matter:

• The crawled content, as HTML or as parsed JSON if you requested a scraper
• cb_status, Crawlbase's verdict on the crawl. 200 means success. It was formerly named pc_status.
• rid, correlating this callback with the dispatch
• original_status, the HTTP status the target site returned
• url, the page that was crawled

The two status fields answer different questions, and conflating them is the most common source of bad data in a pipeline like this. original_status describes what the site said. cb_status describes whether Crawlbase successfully got an answer.

So put a Filter by Zapier step in front of the destination action and continue only when cb_status is 200. Everything downstream then sees pages that were genuinely retrieved.

Keep parsing separate from transport where you can. Formatter by Zapier covers ordinary text work, and Code by Zapier is there when the extraction is genuinely custom. For sites Crawlbase already parses, the scraper parameter returns structured JSON and removes the parsing step altogether.

Verify the whole flow once

Run a single URL end to end before turning on any volume:

1. Confirm the Receiver is On and its Catch Hook URL is what the Dispatcher sends as callback.
2. Turn the Dispatcher On.
3. Fire the trigger: add the test row, run the schedule, submit the form.
4. Check the Dispatcher's history. The Webhooks action should finish quickly and return an rid, not a page.
5. Give the crawl time to complete.
6. Check the Receiver's history for the callback run.
7. Check the destination for the row, the message, or the record.

If the Dispatcher succeeds and the Receiver never runs, there are three usual causes: the callback URL is wrong or has a typo, the Receiver was switched off when Crawlbase tried to deliver, or a Filter step dropped the run before it reached the destination action. Check them in that order.

Cloud Storage as the recovery path

Callbacks are the normal delivery path. Production systems need an answer for the times normal does not happen: the Receiver was mid-edit, a destination app was down, a filter was wrong and dropped a good result.

That is what store=true buys. The response is saved in Crawlbase Cloud Storage against the rid, so a failed callback becomes a retrieval instead of a re-crawl. For occasional recovery, look the crawl up by rid in the storage dashboard. For anything recurring, build a third small Zap:

Storage can also be queried by url rather than rid, which returns the most recently stored version of that page. One limit to design around: stored pages are kept for 14 days by default. Cloud Storage is a recovery buffer, not your archive, so anything you need long term should be copied into your own system by the Receiver.

Production considerations

The pipeline above is a working proof of concept. A handful of additions make it something you can leave running.

Make the Receiver idempotent. Treat rid as a unique key and check whether you have already processed it before writing. Delivery retries, a manually replayed run, or a Zap edited mid-flight can all put the same callback through twice, and a duplicate row is much easier to prevent than to clean up.

Validate before you write. Filter on cb_status, and treat anything else as an exception with a home to go to, whether that is a separate sheet, a Slack alert, or a retry list. Silent failures in an automation are worse than loud ones because nobody looks until the data is already wrong.

Keep the Dispatcher non-blocking. It submits and records. Every time someone adds "just one small step" that reads the body, the timeout returns.

Protect the token. It lives in the Zap configuration, not in screenshots or shared templates. Rotate it if it leaks.

Use the cheapest token that works. The Normal token is faster and costs less than the JavaScript token. Promote a target to the JS token when the Normal response comes back empty or challenge-blocked, not as a precaution.

Respect the target. Check terms of use, robots directives, and rate limits, and stay within them. The automation being easy to build does not change what you are allowed to collect.

Where this pattern fits

The dispatch-and-callback shape is not specific to scraping. It is what you reach for whenever a workflow has to start something slow without waiting for it.

The business logic changes; the execution model does not. A business event starts an asynchronous crawl, a callback delivers the result, and a downstream action does something with it. If you would rather own the receiving end in code than in Zapier, the same architecture in Python is covered in building a Flask callback server.

Conclusion

The problem with synchronous no-code scraping was never the HTTP request. It was tying the lifetime of a workflow step to the lifetime of a web retrieval, two things with no reason to agree on how long they should take.

Separating them removes the dependency. The Dispatcher sends async=true and a callback URL, gets an rid, and finishes in the time an acknowledgement takes. Crawlbase crawls on its own schedule and POSTs the page to the Receiver, which validates it and routes it onward. Adding store=true means the runs that still go wrong are recoverable by rid instead of lost.

What you end up with is a pipeline whose reliability no longer depends on how fast the slowest target feels like being.

Frequently Asked Questions (FAQs)

Can I use Crawlbase with Zapier without writing code?

Yes. The pipeline in this guide uses Webhooks by Zapier to send the request and a Catch Hook to receive the result, with Formatter and Filter steps for processing. No Python and no backend are required. If your targets are fast enough that you do not need async at all, the native Crawlbase Zapier app provides Crawl URL, Scrape Structured Data, and Take Screenshot as ready-made actions.

Why use asynchronous crawling with Zapier instead of a normal request?

Because a Zapier action has a finite execution window and a web retrieval does not. With async=true the dispatching action finishes as soon as Crawlbase accepts the job and returns an rid, so a slow or heavily rendered page no longer decides whether the Zap survives. The finished page arrives separately, at the Receiver.

What is the difference between cb_status and original_status?

cb_status is Crawlbase's verdict on the crawl, where 200 means success. It was formerly named pc_status. original_status is the HTTP status the target site returned. They are independent: a site can answer 404 and still be crawled successfully, and a challenge page can answer 200 while the crawl itself failed. Branch your workflow on cb_status.

What happens if the Zapier callback fails?

If the request included store=true, the response was saved in Crawlbase Cloud Storage against its rid, so you can retrieve it from the storage dashboard or through a small recovery Zap rather than crawling the page again. This is the reason for recording every rid at dispatch time. Stored pages are kept for 14 days by default.

Can I run this pipeline on any website?

Yes. Asynchronous mode works on any domain, so the pipeline is not tied to a particular set of targets. What varies is what each target needs: pages that render client-side want the JavaScript token and sometimes page_wait, and some sites are simply slower than others, which is the problem this architecture exists to absorb. Check the site's terms of use, robots directives and rate limits before pointing a scheduled Zap at it.

NZ

Neil Zamora

Senior Architect · Crawlbase

Senior architect at Crawlbase, focused on the systems behind large-scale crawling: proxy rotation, anti-bot resilience, and the APIs that hide that complexity.