What It Takes to Process 8,000 CAPTCHAs Per SecondThe Concurrency Budget Behind the Number

Eight thousand CAPTCHAs per second is 28.8 million per hour and 691 million per day. A number that size usually gets quoted as a property of a solver, as though the answer were a faster model or a bigger box. It is not. Long before any of that matters, 8,000 solves per second is a concurrency budget, and the size of that budget comes off the back of an envelope.

Throughput equals concurrency divided by service time. If one solve takes two seconds and you want 8,000 of them finishing every second, you need 16,000 of them open at all times. Nothing in that number depends on how clever the solver is. It falls out of the latency you cannot remove.

That reframes the engineering problem. The interesting part is not the solve; it is the control plane around it. Something has to accept work, bound it, keep tens of thousands of operations in flight without falling over, and measure itself well enough that you can name which stage is the limit.

The companion Go implementation in ScraperHub/how-we-solve-8000-captchas-per-second builds that control plane in four files: a bounded queue, a worker pool with backpressure, a pluggable Solver interface with a mock and a Crawlbase implementation, and a metrics collector that reports solves per second alongside p50 and p99. This post walks through it, runs both benchmarks, and then does the honest accounting on what those runs do and do not prove about 8,000.

Throughput is concurrency divided by service time

Little's Law is the whole capacity plan in one line. The number of operations in flight equals the completion rate multiplied by how long each one takes. Rearranged for the question at hand: the rate you can sustain is your concurrency divided by your service time.

This is worth checking against the repository's own numbers rather than taking on faith. The documented mock run uses 256 workers and reports a p50 of 7.83 ms. Divide: 256 workers over 7.83 ms per solve predicts 32,695 solves per second. The run measured 32,507. The queue, the results channel, the fan-in collector and its mutex together cost 0.57% of the theoretical ceiling.

Run the same arithmetic in the other direction and the title of this post turns into a hardware order.

The two rows tell you where the difficulty actually lives. Simulated work is 250 times faster per item than a real network round trip, so the local benchmark needs three orders of magnitude less concurrency to post a bigger number. Everything hard about 8,000 real solves per second is the 16,000 open sockets, not the 8,000 solves.

Four files, one interface

The canonical runnable module is the final/ directory of the companion repository. It is deliberately small.

pipeline.go   bounded queue, worker pool, results fan-in
solver.go     the Solver interface and its two implementations
metrics.go    throughput and latency percentiles
main.go       load harness and flags
config.go     token and target URL from the environment

The Solver interface is the seam that makes the rest of it testable. The pipeline never learns whether a challenge went to a local simulation or across the internet; it submits work and records what came back. That is what lets you benchmark the orchestration at 32,000 operations per second without a network, then point the identical pipeline at a live endpoint and watch the numbers collapse for reasons you can name.

Step 1: the queue and the worker pool

A challenge is a struct and the queue is a buffered channel. Channel capacity is the entire backpressure mechanism: once the buffer is full, the producer's send blocks until a worker frees a slot, so an overloaded system slows its intake instead of growing its heap.

Source: final/pipeline.go

func NewPipeline(workers, queueSize int, solver Solver, metrics *Metrics, target string) *Pipeline {
    return &Pipeline{
        workers: workers,
        queue:   make(chan Challenge, queueSize), // bounded => backpressure
        results: make(chan Result, queueSize),
        solver:  solver,
        metrics: metrics,
        target:  target,
    }
}

Each worker is a goroutine ranging over that channel. Ranging over a channel is also the shutdown protocol: the producer closes the queue when it runs out of work, the range loops drain what is left and then exit, and sync.WaitGroup tells the caller when the last one is done.

func (p *Pipeline) worker(ctx context.Context, id int, wg *sync.WaitGroup) {
    defer wg.Done()

    for ch := range p.queue {
        started := time.Now()

        err := p.solver.Solve(ctx, ch)

        p.results <- Result{
            ID:      ch.ID,
            OK:      err == nil,
            Latency: time.Since(started),
            Worker:  id,
        }
    }
}

Note what the worker does not do: it does not touch the metrics struct. It reports a Result and moves on. A single collector goroutine drains the results channel into the collector, so the counters have exactly one writer and the hot path stays a channel send.

Step 2: the solver interface and its two implementations

The pipeline depends on two methods. That is the whole contract.

Source: final/solver.go

type Solver interface {
    Solve(ctx context.Context, c Challenge) error
    Name() string
}

MockSolver stands in for the solve stage with a base latency, uniform jitter, and a failure rate, and makes no network calls. It exists so you can measure the orchestration on its own.

func (s *MockSolver) Solve(ctx context.Context, _ Challenge) error {
    d := s.Base

    if s.Jitter > 0 {
        d += time.Duration(rand.Int63n(int64(s.Jitter)))
    }

    select {
    case <-time.After(d):
    case <-ctx.Done():
        return ctx.Err()
    }

    if s.FailRate > 0 && rand.Float64() < s.FailRate {
        return errors.New("mock solve failed")
    }

    return nil
}

CrawlbaseSolver is the real path. It sends the target URL to the Crawling API, where the CAPTCHA and anti-bot handling happen as part of the fetch, and treats a clean 200 with a fully drained body as a completed operation. Draining to io.Discard is not cosmetic: an unread body cannot be returned to the connection pool, which matters a great deal in a moment.

func (s *CrawlbaseSolver) Solve(ctx context.Context, c Challenge) error {
    endpoint := fmt.Sprintf(
        "https://api.crawlbase.com/?token=%s&url=%s",
        url.QueryEscape(s.Token),
        url.QueryEscape(c.URL),
    )

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
    if err != nil {
        return err
    }

    resp, err := s.Client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    _, _ = io.Copy(io.Discard, resp.Body)

    if resp.StatusCode != http.StatusOK {
        return fmt.Errorf("crawlbase status %d", resp.StatusCode)
    }

    return nil
}

Same signature, wildly different physics. The worker pool cannot tell them apart, which is the point.

Step 3: measure throughput and the tail together

A throughput number on its own hides the interesting failures. A pipeline can post a healthy average while a slice of requests takes an order of magnitude longer than the rest, and in a concurrent system those slow requests are also the ones holding your concurrency budget hostage. So the collector keeps every latency and computes percentiles from them.

Source: final/metrics.go

func (m *Metrics) Report() Report {
    total := m.successes + m.failures
    elapsed := m.end.Sub(m.start)

    rate := 0.0
    if elapsed > 0 {
        rate = float64(total) / elapsed.Seconds()
    }

    return Report{
        Total:           total,
        Successes:       m.successes,
        Failures:        m.failures,
        Elapsed:         elapsed,
        SolvesPerSecond: rate,
        P50:             m.percentile(50),
        P99:             m.percentile(99),
    }
}

Four signals come out of this: total volume, the success and failure split, p50, and p99. The pair is what makes worker-count tuning a measurement instead of a guess. Add workers and watch both: if the rate climbs while p99 holds, the extra concurrency is being absorbed. If the rate flattens while p99 keeps rising, you are queueing somewhere downstream and adding workers is now making things worse.

Step 4: the load harness

main.go wires the pieces together and puts the dials on the command line, while config.go reads the token and target URL from the environment through a small .env loader with no dependencies.

Source: final/main.go

metrics := NewMetrics(*requests)

pipeline := NewPipeline(
    *workers,
    *queueSize,
    solver,
    metrics,
    cfg.TargetURL,
)

report := pipeline.Run(
    context.Background(),
    *requests,
)

fmt.Println(report)

The flags are -solver, -requests, -workers, -queue, -base-ms, -jitter-ms, and -fail-rate. Defaults are 20,000 requests, 256 workers, a queue of 1,024, a 5 ms base latency with 5 ms of jitter, and a 1% failure rate.

What the mock run proves, and what it cannot

Load-test the control plane first, with no network in the way:

go run . -requests 20000 -workers 256 -queue 1024

solver=mock requests=20000 workers=256 queue=1024 target=https://example.com

total=20000  ok=19804  fail=196  elapsed=615ms
solves/sec=32507  p50=7.83ms  p99=10.354ms

Three things in that output are worth reading closely, because two of them are confirmations and one is a limit.

32,507 against a ceiling of 32,695. The orchestration costs 0.57%. Channel sends, one mutex, and a single-goroutine fan-in are not where a throughput problem lives, and now you have the receipt rather than the intuition.

196 failures out of 20,000 is 0.98%, against a configured failure rate of 1%. The error path is being exercised and counted correctly. Pass -fail-rate 0 for a clean run, but a solver that never fails is not the one you are shipping.

p99 of 10.354 ms is not a tail. The mock's service time is 5 ms plus a uniform jitter under 5 ms, so 10 ms is a hard arithmetic maximum. The measured p99 sits 354 microseconds above a ceiling built into the simulation. That is a scheduling measurement, not a latency distribution. Real tails are made of DNS, TLS handshakes, retries, a slow origin, and one unlucky IP, and none of those exist in this run. Believing a p99 from a mock is the single easiest way to be surprised in production.

The real path is bounded by the network

Now the same pipeline, same code, against a live endpoint. Small on purpose:

go run . -solver crawlbase -requests 12 -workers 6 -queue 32

solver=crawlbase requests=12 workers=6 queue=32

total=12  ok=12  fail=0  elapsed=3.38s
solves/sec=4  p50=1.924091s  p99=2.409689s

Four solves per second, and the reported figure is 3.55 rounded by the format string. Twelve samples is far too few for a meaningful percentile, so treat the p50 as an order of magnitude: a solve through the real path takes roughly two seconds, most of which is the fetch and the anti-bot work at the far end rather than anything happening in Go.

That single number is the one that matters for capacity planning, and it is the one the mock cannot give you. Two seconds of service time is what turns 8,000 per second into 16,000 concurrent operations. The comparison is not mock 32,507 versus real 4; it is a control plane with three orders of magnitude of headroom sitting in front of a dependency that sets the actual budget.

Three things break between 6 workers and 16,000 in flight

The sample runs at 6 workers. Production runs at four figures. These are the three walls you hit on the way, in the order you hit them.

The connection pool goes first

NewCrawlbaseSolver builds its client the ordinary way:

Client: &http.Client{Timeout: 30 * time.Second}

No Transport field means http.DefaultTransport, and http.DefaultTransport keeps two idle connections per host. That is DefaultMaxIdleConnsPerHost, and it has been 2 for as long as net/http has existed. At six workers against one API host, nobody notices. At two thousand workers against one API host, all but two connections are torn down as soon as each response is read, so nearly every solve pays for a fresh TCP handshake and a fresh TLS handshake before it can send a byte. You have added a round trip or two to a two second operation, burned CPU on handshakes, and started cycling through ephemeral ports for no reason.

Size the transport for the concurrency you actually want:

transport := http.DefaultTransport.(*http.Transport).Clone()
transport.MaxIdleConns = workers
transport.MaxIdleConnsPerHost = workers // default is 2
transport.MaxConnsPerHost = workers    // 0 means unlimited
transport.IdleConnTimeout = 90 * time.Second

client := &http.Client{Transport: transport, Timeout: 30 * time.Second}

Setting MaxConnsPerHost is also how you make backpressure honest end to end. Without it, a worker pool of 16,000 will happily try to open 16,000 sockets. With it, workers block waiting for a connection instead, which is a much better failure mode than exhausting file descriptors. Raise the process limit too, because the operating system default is nowhere near four figures of concurrent sockets.

The latency slice turns into a memory leak

The collector appends one time.Duration per solve and preallocates the slice to the request count. For a 20,000 request benchmark that is 160 KB and a sort you never notice. For a service, it never stops growing.

At 8,000 solves per second, eight bytes per sample is 62.5 KB per second, 230 MB per hour, and 5.5 GB per day. Worse, percentile copies the whole slice and sorts the copy on every call, so a report over one hour of traffic sorts 28.8 million elements and doubles the footprint while it does it. That is correct behavior for a load harness, whose whole job is to keep every sample from a bounded run, and completely wrong for anything long-lived.

The fix is the standard one: a fixed-size latency histogram with bucketed counts, reset or rotated per reporting window. Percentiles become an interpolation over counters instead of a sort over history, memory becomes constant, and the numbers stop lying to you the moment a run outlives its allocation.

One process stops being the deployment

Sixteen thousand in-flight requests is not a goroutine count problem. Goroutines are cheap; the sockets, the TLS sessions, the file descriptors, and the NIC in front of them are not. Past a few thousand concurrent connections to one destination, the shape has to change: many worker instances, one shared queue in front of them, results streaming out to somewhere durable.

                    +----------------+
                    |  shared queue  |   bounded, same as the channel
                    +-------+--------+
                            |
             +--------------+--------------+
             |              |              |
             v              v              v
        worker group   worker group   worker group
             |              |              |
             +--------------+--------------+
                            |
                            v
                     solve path (API)
                            |
                            v
                      result stream

The model survives the move because nothing in it assumed a single process. A Go channel becomes a shared queue, a goroutine becomes a worker instance, the fan-in becomes a metrics pipeline, and queue to workers to solver to results reads the same at both scales. That is the real argument for keeping the control plane this small: it is the same four responsibilities whether you run 256 workers or 8 instances of 2,000. The same reasoning shows up in more detail in our write-up on building a distributed crawling engine.

Queue size and worker count do different jobs

These two flags get tuned together and confused constantly.

Worker count sets concurrency. It decides how many solver operations you are willing to have open at once, which by Little's Law is the only lever that changes your sustainable rate.

Queue size sets buffering. It decides how much temporary imbalance between arrivals and capacity you can absorb before the producer starts blocking.

A bigger queue buys zero throughput. If workers can finish 1,000 operations per second and producers generate 2,000, then queue depth only chooses how many seconds pass before the queue is full. It fills, the producer blocks, and backpressure arrives exactly where it should. Sizing the queue is really a decision about burst tolerance and how stale you are willing to let a queued item get.

Tune the mock first, where iteration is free: -base-ms changes the simulated service time, -jitter-ms adds variance, and -fail-rate lets you watch the error path under load. Those three dials reproduce most of the behaviors you care about before a single real request is sent.

What to carry into production

Bound the work

An unbounded queue does not prevent overload; it hides overload until memory becomes the failure mechanism, and then fails all at once with no useful signal. A bounded queue converts the same overload into backpressure, which is visible, survivable, and measurable.

Keep the solver behind an interface

The pipeline should move work, not hold opinions about how work gets done. The two implementations here are the argument: the same control plane got load-tested at 32,507 operations per second and then pointed at a live API without a line changing in pipeline.go.

Measure the stage, not the system

Throughput and tail latency only mean something together, and only when you know which stage produced them. The worker pool can never expose more useful throughput than the solver sustains, so a rate that stops responding to added workers is a statement about the dependency. Our notes on scaling web scraping projects and this 1 billion requests per month case study both come back to the same habit of instrumenting per stage.

Running the companion repository

The repository needs Go 1.22 or newer. The mock path needs no account; the Crawlbase path needs a token.

git clone https://github.com/ScraperHub/how-we-solve-8000-captchas-per-second.git
cd how-we-solve-8000-captchas-per-second/final
cp .env.example .env        # only needed for the crawlbase solver
go build -o captcha-pipeline .

Two environment variables, both read by config.go:

final/ is the canonical runnable module and steps/ holds read-only checkpoint copies of the file introduced at each step above, so you can read the pipeline as it was after step 1 rather than only in its finished form.

Conclusion

What it takes to process 8,000 CAPTCHAs per second is 16,000 operations in flight, and every hard part follows from that one figure rather than from the solving itself.

The Go control plane is the easy half, and the measurements say so: a bounded queue, a worker pool, a two-method interface, and a fan-in collector run within 0.6% of their arithmetic ceiling at 32,507 simulated operations per second. The hard half is holding four figures of real connections open, keeping instrumentation constant in memory while doing it, and spreading the whole thing across instances once one machine runs out of sockets.

So the useful output of this exercise is not a worker count. It is two numbers you can defend, service time and concurrency, and a pipeline instrumented well enough to tell you which stage owns the limit. Get those and the target rate becomes a capacity decision. Skip them and it stays a guess with a big number attached.

Frequently asked questions

Does the sample command solve 8,000 real CAPTCHAs per second?

No, and it is not meant to. The mock command reports 32,507 simulated solves per second with no network involved, and the Crawlbase command in the repository uses a deliberately tiny 12 request workload that reports about 4 per second because it is network-bound. The 8,000 figure describes the architecture at production scale, which means many worker instances against a shared queue, not one local process. What the sample gives you is the two inputs you need to size that deployment: the control plane's headroom and the real path's service time.

How many workers do I need for 8,000 solves per second?

Divide the target rate by the completion rate of a single worker. At roughly two seconds per solve, one worker finishes 0.5 per second, so 8,000 per second needs about 16,000 workers in flight. Whether that is 8 instances of 2,000 or 16 of 1,000 is a question about sockets, descriptors, and blast radius, not about Go. Measure your own p50 against your own targets before trusting the two second figure.

Why benchmark with a mock solver at all?

To find out whether your own code is the constraint before you start blaming a dependency. The mock removes the network and simulates the solve with configurable latency, jitter, and failure rate, which isolates the queue, the worker pool, and the fan-in. Here it showed the orchestration costs 0.57% of its theoretical ceiling, so a throughput shortfall in the real run is provably not in the control plane.

Can I trust the p99 from a mock run?

No. The mock's service time is a base latency plus bounded jitter, so its worst case is fixed by arithmetic: 5 ms plus at most 5 ms equals a 10 ms ceiling, and the measured p99 of 10.354 ms is that ceiling plus scheduling overhead. A real tail comes from DNS, TLS, retries, and slow origins, none of which the mock simulates. Use mock percentiles to detect scheduling problems in your own code and nothing else.

Why a bounded queue instead of an unbounded one?

Because an unbounded queue does not remove the overload, it just relocates the symptom to memory and delays the failure until it is unrecoverable. When a bounded queue fills, producers wait for workers, which is backpressure: the pressure propagates back to whoever is generating work, while pending work stays under a limit you chose deliberately.

How should I tune the worker count?

Raise -workers on the mock path and watch throughput and p99 together. While the rate climbs and p99 stays flat, the added concurrency is being absorbed. When the rate flattens and p99 keeps climbing, the downstream stage has become the limit and more workers only deepen a queue you cannot see. Then switch to the real solver and repeat, because the two curves are shaped nothing alike.

What does the Crawlbase solver actually do?

It sends the target URL to the Crawling API and treats a clean HTTP 200 with a drained body as a completed solve. The CAPTCHA and anti-bot work happens inside that fetch rather than in your process, which is why a "solve" on this path is one request and one status check. Approaches to the same problem from the caller's side are covered in our guide to bypassing CAPTCHAs in web scraping.

NZ

Neil Zamora

Senior Architect · Crawlbase

Senior architect at Crawlbase, focused on the systems behind large-scale crawling: proxy rotation, anti-bot resilience, and the APIs that hide that complexity.