Building an LLM-Ready Stack Exchange Corpus33 Million Threads with the Crawling API

We recently built a complete question-and-answer corpus of the Stack Exchange network: every site, every question, every answer, every comment, as training data for a customer's LLM work. About 33 million threads across 360+ sites, delivered as JSONL with every post body converted from HTML to Markdown.

This post walks through the engineering. How we modeled the network as fetchable units and sized the job before committing to it, the data-quality traps that only show up with real data, a content-licensing bug we shipped in an early sample and had to disprove empirically, and where the Crawling API earns its keep in a job like this. The numbers are real measurements from the collection work; client details are removed.

The deliverable: complete threads, not pages

The first design decision is the unit of delivery. Pages are what you fetch; they are not what anyone wants to consume. The natural unit for Q&A data is the assembled thread: one record containing the question, all of its answers, and all comments on both, with the metadata that lets a consumer filter and weight (scores, tags, timestamps, accepted-answer markers).

One thread per JSONL line:

{
  "site": "stackoverflow.com",
  "question_id": 11227809,
  "title": "Why is conditional processing of a sorted array faster...",
  "tags": ["java", "c++", "performance", "cpu-architecture"],
  "score": 27538,
  "view_count": 1990935,
  "answer_count": 26,
  "accepted_answer_id": 11227902,
  "is_answered": true,
  "creation_date": "2012-06-27T13:51:36Z",
  "last_edit_date": "2026-04-08T09:20:07Z",
  "content_license": "CC BY-SA 4.0",
  "closed_date": null,
  "owner": {"user_id": 87234, "display_name": "GManNickG", "reputation": 507167},
  "body_markdown": "In this C++ code, sorting the data *(before the timed region)*...",
  "comments": [ ... ],
  "answers": [
    {
      "answer_id": 11227902,
      "link": "https://stackoverflow.com/a/11227902",
      "is_accepted": true,
      "score": 35287,
      "content_license": "CC BY-SA 4.0",
      "owner": { ... },
      "body_markdown": "**You are a victim of [branch prediction](...) failure.**...",
      "comments": [ ... ]
    }
  ]
}

Four design rules saved us later:

• No silent truncation, ever. Threads are delivered whole; the largest one we validated had 105 answers. If you must subset anything (we did, in early format samples), carry explicit *_total and *_included fields. A data customer will diff your sample against the live site; make sure what they find is documented behavior, not a surprise.
• Comments attach at both levels. Question comments and per-answer comments are different lists. In our first format sample we shipped question comments only. Of the roughly 1,050 comments in an 8-thread sample, more than 900 were on answers. Easy to miss, embarrassing to be told.
• Deleted users get a defined placeholder ({"user_id": null, "display_name": "[deleted]", "reputation": null}), not a null-ish soup that every consumer handles differently.
• Every answer gets a permalink. https://<site>/a/<answer_id> is stable and cheap to construct; attribution and spot-checking both need it.

Sizing the job: the request model

Everything is collected through the Crawling API: a single GET per page, with proxy rotation, retries, and JavaScript rendering (when needed) handled behind the endpoint.

import requests

CRAWLBASE = "https://api.crawlbase.com/"
TOKEN = "YOUR_TOKEN"

def fetch(url):
    r = requests.get(CRAWLBASE, params={"token": TOKEN, "url": url}, timeout=90)
    r.raise_for_status()
    return r.text

Stack Exchange question pages are server-rendered HTML, so the standard token (no JavaScript rendering) is enough, and that keeps the job fast and cheap. A full-network collection breaks down into four kinds of fetch units:

Total: roughly 38 to 50 million requests. The one soft line is comment expansions. You cannot know how many posts have hidden comments until you look, so we sized it as a range and, on the commercial side, put a cap on the estimate rather than pretending to precision we did not have. Sizing the job before running it, with stated uncertainty, is most of what separates a quote you can stand behind from a guess.

Why a proxy layer at all? Stack Exchange is not aggressively anti-bot, but any single-IP crawler hits throttling long before 33 million pages. During scoping we hit 429s within the first 55 or so rapid requests from one IP. At production scale you need rotation, backoff, and retry accounting, and that is precisely the layer the Crawling API replaces.

HTML to Markdown, and the traps

LLM teams want Markdown, not HTML. The conversion sounds trivial and is not. Three traps we hit with real data:

Trap 1: naive regex converters eat code

Stack Exchange bodies are full of <pre><code> blocks containing lines like #include <iostream>, and <iostream> looks exactly like an HTML tag to a strip-tags regex. Our first quick-and-dirty converter silently deleted the includes from one of the most famous C++ questions on the site. Use a real DOM-based converter (we used markdownify), then verify fidelity on code-heavy samples specifically:

from markdownify import markdownify as md

def to_markdown(html):
    return md(html, heading_style="ATX", bullets="-").strip()

Trap 2: nested-link artifacts

Anchor tags wrapping other markup can serialize as [text]([label](url)). We scan for the pattern and collapse it in post-processing:

import re

NESTED = re.compile(r'\[([^\[\]]*)\]\(\s*\[([^\[\]]*)\]\(([^()\s]+)\)\s*\)')
while True:
    fixed = NESTED.sub(lambda m: f"[{m.group(1) or m.group(2)}]({m.group(3)})", text)
    if fixed == text: break
    text = fixed

Then assert zero occurrences across the whole corpus before shipping. "We fixed the converter" is a claim; a corpus-wide scan returning zero is a fact.

Trap 3: don't "fix" LaTeX

Math sites (math.stackexchange, stats, physics) carry LaTeX inline: $\frac{\textrm{d}y}{\textrm{d}x}$. Preserve it byte for byte. It is valid, dense, and exactly what models train well on. The mistake is a well-meaning cleanup pass that strips "weird symbols". Make LaTeX preservation an explicit, stated property of the corpus and test for it.

The licensing bug: keyed to the wrong date

Every Stack Exchange post is Creative Commons licensed, and the license version depends on when the content was contributed:

We attach content_license to every question, answer, and comment, together with author attribution and a permalink: the pieces CC BY-SA attribution needs. For a handful of very old posts the license is not present in the source data, so we derived it from the date table. Our first pass keyed the derivation to the post's creation date. A sharp-eyed reviewer caught a 2008 question, edited in 2023, tagged CC BY-SA 2.5, and asked which date the rule actually uses.

Instead of arguing from documentation, we tested it against the data. Across 1,384 posts that do carry a source license, we checked both hypotheses. All 331 source-licensed posts in the validation slice matched the last-edit rule, including all 137 posts created in one license era and edited in another, of which exactly zero matched creation-date keying. The license belongs to the current revision, so it keys to the last-edit date, falling back to creation date for never-edited posts.

Two lessons. First, the technical one: derive licenses from last_edit_date or creation_date, never creation_date alone. Second, the meta one: when a data consumer challenges a field's semantics, the answer that ends the conversation is an empirical test over the corpus, not a citation.

Semantics you must document

Fields that look self-explanatory and are not:

• is_answered does not mean "has an accepted answer." It is true when the question has an accepted answer or any positively scored answer. If you do not document this, someone downstream will misread it. We also added accepted_answer_id (nullable) at the question level so "has an accepted answer" needs no scanning of the answers array.
• View counts on pages are rounded ("Viewed 2.0m times"); exact integers come from structured sources. If your pipeline mixes both, state which one a field carries.
• The corpus is alive. Posts are edited, deleted, closed, and protected continuously. Deliver the state flags (closed_date, closed_reason, protected_date, locked_date) and timestamp everything in ISO-8601 UTC, so consumers can reason about snapshot-time truth.

QA before shipping

Every claim above turns into an assertion that runs against the shipped artifact:

import json

threads = [json.loads(l) for l in open("threads.jsonl")]

assert all(len(t["answers"]) == t["answer_count"] for t in threads)          # complete
assert all(a["link"] and a["content_license"] for t in threads
           for a in t["answers"])                                            # attribution
assert all(c["creation_date"] for t in threads
           for a in t["answers"] for c in a["comments"])                      # timestamps
assert not any(NESTED.search(t["body_markdown"]) for t in threads)            # conversion
acc = lambda t: [a["answer_id"] for a in t["answers"] if a["is_accepted"]]
assert all((t["accepted_answer_id"] is None and not acc(t))
           or acc(t) == [t["accepted_answer_id"]] for t in threads)           # consistency

The pattern that served us best across the whole project: ship a small sample early, let the consumer diff it against the live site, and treat every discrepancy they find as a schema improvement. Our format went through three sample iterations before a byte of the full corpus was collected, which is far cheaper than discovering the same issues after 33 million threads.

Takeaways

• Model the site as fetch units and size the job (with stated uncertainty) before you run it.
• Deliver domain units (threads), not fetch units (pages), and never truncate silently.
• HTML to Markdown is a fidelity problem, not a formatting nicety: DOM-based conversion, artifact scans, LaTeX left alone.
• Attach license, attribution, and a permalink to every post, and key derived licenses to the last-edit date. Validate semantics empirically when challenged.
• Document the fields that lie (is_answered), and assert everything you promised against the artifact you actually ship.

The collection layer for all of this (rotation, retries, rendering, success-only accounting) is the Crawling API; the corpus engineering on top is the part that is genuinely yours. If your pipeline needs to survive past a prototype, our guide on scaling web scraping projects covers the production concerns, and if the destination is an AI workflow rather than a JSONL handoff, see how we build research datasets with the Web MCP Server.

Frequently asked questions

How many requests does it take to collect all of Stack Exchange?

Roughly 38 to 50 million for a complete pass: about 0.7 million listing pages for discovery, 33 million question pages, a few hundred thousand extra answer pages for threads with more than 30 answers, and 4 to 16 million comment-expansion calls. The expansion count is the one genuine unknown before you crawl, so size it as a range with a cap instead of a false point estimate.

Why deliver assembled threads instead of raw pages?

Because consumers train on conversations, not on HTML documents. One JSONL record per thread, containing the question, every answer, and every comment with scores, tags, timestamps, and license, means a consumer can filter and weight without reassembling state themselves. Pages are a fetch-time concept; they should be gone by delivery time.

How do you keep CC BY-SA attribution correct at corpus scale?

Attach three things to every post: the content_license (keyed to the last-edit date, falling back to creation date for never-edited posts), the author with a defined placeholder for deleted users, and a stable permalink. Then validate the license derivation empirically against posts that carry an explicit source license rather than trusting documentation.

Does converting HTML to Markdown lose information?

It can, in ways that matter for training. Strip-tags regexes delete C++ includes because <iostream> looks like markup, nested anchors serialize as broken link syntax, and overeager cleanup passes mangle LaTeX. Use a DOM-based converter, scan the whole corpus for known artifact patterns, and preserve LaTeX byte for byte.

VP

Viktor Petrov

Data Science Architect · Crawlbase

Data Science Architect at Crawlbase. Works on the models that keep large-scale crawling reliable against modern bot defenses.