Build a Knowledge Graph from Web DataCrawl Company Sites into Neo4j with an LLM

A folder of scraped company pages tells you what each website says about itself. Answering a question across those pages takes one more step: recognising that the same people, tools and cities appear on several sites, and connecting them to the companies that mention them. Which companies share an integration? Who is named on more than one site? Where do their offices overlap?

This post builds a Python pipeline that discovers the relevant pages of a company site with the Crawling API, fetches them as Markdown, asks an LLM to extract entities and relationships into a fixed schema, checks that every supporting quote is really on the page, normalises names, and loads the result into Neo4j with MERGE. Cypher then answers the cross-company questions, and a small GraphRAG layer turns a plain-English question into a query and an answer that cites its source pages.

The code lives in ScraperHub/build-a-knowledge-graph-from-crawled-websites-with-neo4j: nine walkthrough scripts under steps/, one per transformation, and the packaged CLI with caching, Docker configuration and tests under final/. The numbers below come from recorded runs against Basecamp, HEY, Pipedrive and Zendesk on 29 October 2026.

Model the relationships you want to query

A knowledge graph stores entities as nodes and connects them with typed relationships. Here a Person can WORKS_AT a Company, and a Company can INTEGRATES_WITH an Integration. Relational tables can hold the same facts; the reason to use Neo4j is that a traversal becomes a visible pattern. (a:Company)-[:INTEGRATES_WITH]->(i)<-[:INTEGRATES_WITH]-(b:Company) reads as "two companies connected to the same integration", and it stays readable as the questions add hops.

The vocabulary is deliberately small: five labels (Company, Person, Product, Integration, Location) and five relationship types (WORKS_AT, OFFERS, INTEGRATES_WITH, PARTNERS_WITH, LOCATED_IN). A small vocabulary is easier to validate at extraction time and easier to constrain when a model writes Cypher later.

Prepare the walkthrough and the database

You need Python 3.11+, Docker with Compose, a Crawlbase account with its Normal and JavaScript tokens, and an OpenAI API key. The repository uses gpt-4o-mini with Pydantic structured outputs and runs Neo4j 5 Community locally; dependency versions are pinned in steps/requirements.txt. Clone it, create a virtual environment and copy the two environment templates:

git clone https://github.com/ScraperHub/build-a-knowledge-graph-from-crawled-websites-with-neo4j.git
cd build-a-knowledge-graph-from-crawled-websites-with-neo4j
python -m venv .venv
source .venv/bin/activate
cp steps/.env.example steps/.env
cp final/.env.example final/.env

Fill in both .env files with the same Neo4j password before starting the database. The walkthrough scripts read steps/.env; Docker Compose and the packaged CLI read final/.env. Both use the same keys:

CRAWLBASE_TOKEN=YOUR_CRAWLBASE_TOKEN
CRAWLBASE_JS_TOKEN=YOUR_CRAWLBASE_JS_TOKEN
OPENAI_API_KEY=YOUR_OPENAI_API_KEY
OPENAI_MODEL=gpt-4o-mini
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=YOUR_NEO4J_PASSWORD

Then install the walkthrough dependencies and start Neo4j:

pip install -r steps/requirements.txt
cd final && docker compose up -d --wait

Run the scripts in numerical order from steps/. Every script writes its output as JSON under steps/.cache/, and the later ones read what the earlier ones wrote. Use a dedicated tutorial database: 06_load_into_neo4j.py clears all existing graph data before it demonstrates two consecutive loads. Neo4j Browser is at http://localhost:7474.

Discover company pages from homepage links

Discovery requests each homepage with scraper=generic-extractor, one of the Crawlbase scrapers, and reads the links it returns. One finding from the recorded runs is worth knowing before you write code against the docs: the documented schema lists fields such as main_content and headings, while the live responses carried title, meta, content, images and links, among others. The SDK exposes the parsed page as response["json"]. Across five recorded homepages, 422 of 424 links were already absolute HTTP(S) URLs; the other two were mailto: and tel:. The code still resolves relative links with urljoin and drops non-page schemes, so it keeps working if the shape changes.

The first script keeps same-host links whose last path segment matches a company-page pattern:

PAGE_TYPES = {
    "about": r"^(about|company|our-story)",
    "team": r"^(team|our-team|leadership|people)",
    "partners": r"^partner",
    "integrations": r"^(integration|marketplace)",
    "careers": r"^(career|jobs)",
}
MAX_PAGES = 8


def host(url: str) -> str:
    return (urlsplit(url).hostname or "").lower().removeprefix("www.")


def page_type(url: str) -> str | None:
    segments = [s for s in urlsplit(url).path.lower().split("/") if s]
    for name, pattern in PAGE_TYPES.items():
        if segments and re.match(pattern, segments[-1]):
            return name
    return None

Matching the final segment means /company/legal is not an about page just because its parent folder says company. The limit applies to discovered pages; the homepage is fetched separately. This is bounded homepage-link discovery: pages behind pagination, deeper navigation or a careers subdomain need extra rules.

The SDK call passes both the seed homepage and the URL the response actually landed on into discovery, and the two serve different purposes. In a recorded Zendesk request the site redirected to zendesk.de while its navigation still linked to zendesk.com; filtering on the final host discarded every relevant link, filtering on the seed host kept them, and the final URL stayed the base for resolving relative paths. Crawlbase picks the exit country for you, so handle redirects explicitly rather than assuming the landing host. The walkthrough produced:

Basecamp: 52 links, 2 company pages (landed on https://basecamp.com/)
  about         https://basecamp.com/about
  integrations  https://basecamp.com/integrations
HEY: 54 links, 0 company pages (landed on https://www.hey.com/)
Pipedrive: 103 links, 4 company pages (landed on https://www.pipedrive.com/)
  integrations  https://www.pipedrive.com/en/marketplace
  about         https://www.pipedrive.com/en/about
  careers       https://www.pipedrive.com/en/jobs
  partners      https://www.pipedrive.com/en/partner
Zendesk: 125 links, 4 company pages (landed on https://www.zendesk.com/)
  integrations  https://www.zendesk.com/marketplace
  ...

HEY's homepage matched no company-page URL in this run, so its homepage alone went forward. That describes the rule's coverage, not the site. The packaged discovery module also sorts shallow paths first and spreads its selections across page types.

Fetch Markdown and keep the input behind each fact

Extraction needs the actual pages, including the team grids and integration directories that a homepage summary omits. The fetch stage requests format=md with store=true, and keeps the rid under which Cloud Storage holds the Markdown next to the text itself.

Readability filtering trims navigation and boilerplate, but the recorded calls show it can also remove exactly the structured sections this graph needs:

So the fetch step tries readability first, falls back to plain Markdown, and finally to the JavaScript token:

def fetch_page(url: str) -> dict:
    """Readability first, plain Markdown if that drops the content, JS token if the page renders in the browser."""
    for token, readability, min_chars in ((CRAWLBASE_TOKEN, True, 1500),
                                          (CRAWLBASE_TOKEN, False, 300),
                                          (CRAWLBASE_JS_TOKEN, False, 0)):
        response = fetch(url, token, readability)
        if len(response.text) >= min_chars:
            break
    rid = httpx.URL(response.headers.get("storage_url", "")).params.get("rid", "")
    return {"url": url, "markdown": response.text, "rid": rid, "readability": readability,
            "js": token == CRAWLBASE_JS_TOKEN}

The 1,500- and 300-character thresholds are application heuristics: a long navigation shell can pass them without the data you want, and a short legitimate page can trigger a needless retry. For targets you already know render in the browser, the packaged CLI accepts use_js=true in the seed CSV and starts with the JavaScript token. The recorded comparisons show why rendering should be target-specific: Markdown length was identical with both tokens for Linear, Monday and Figma and close for Notion, and Linear's integrations page returned 24,218 characters either way, while quotes.toscrape.com/js returned 42 characters with the Normal token and 887 with the JavaScript one. Modern company sites tend to render on the server; that does not prove every site does.

Two details of the response matter for provenance. For a format=md request with store=true, the rid arrives inside the storage_url response header rather than as a header of its own, and that URL contains the account token, so the code keeps only the rid and never logs the URL. It also uses httpx for these fetches because the pinned Crawlbase Python SDK keeps only original_status, cb_status and url from a response and drops storage_url. Each fallback is a separate request with its own rid, so the rid you keep must belong to the Markdown you actually extract from:

Basecamp     3253 chars  readability  rid=abac8926164450fab0456a27  https://basecamp.com/about
Basecamp    16676 chars  plain        rid=86c17c449811ab0e0f861dd3  https://basecamp.com/integrations
Pipedrive   17742 chars  plain        rid=d2ef101d7ac8d30bf3a26a5a  https://www.pipedrive.com/en/marketplace
Pipedrive   11187 chars  plain        rid=d8d8e67054e10a90c81fa2cb  https://www.pipedrive.com/en/about
Zendesk      8267 chars  plain        rid=78b15c345cde1256878b45bd  https://www.zendesk.com/company

One recorded request returned HTTP 520 with cb_status: 0 and a body saying the call was not charged; a retry succeeded. The packaged crawl client adds transport retries and request pacing, but a long error body is not page content. Check the status before you extract.

Define the extraction contract before calling the model

For the graph to be queryable, the model's output needs a fixed vocabulary and predictable fields. It fills a Pydantic schema that maps one-to-one onto the labels and relationship types, and every entity and relationship carries a source_quote for the check that follows:

class Evidence(BaseModel):
    model_config = ConfigDict(extra="forbid")
    name: str = Field(min_length=1)
    source_quote: str = Field(min_length=3, description="Exact words copied from the page.")


class Company(Evidence): pass
class Product(Evidence): pass
class Integration(Evidence): pass


class Person(Evidence):
    title: str | None
    employer: str | None


class Location(Evidence):
    country: str | None


class Relationship(BaseModel):
    model_config = ConfigDict(extra="forbid")
    source: str
    type: Literal["WORKS_AT", "OFFERS", "INTEGRATES_WITH", "PARTNERS_WITH", "LOCATED_IN"]
    target: str
    source_quote: str = Field(min_length=3)

The full schema module also defines the top-level lists and demonstrates the contract by parsing a valid payload and rejecting an invalid one:

Good output parses: Paulo Cunha
Bad output rejected with 2 errors:
   people.0.source_quote -> Field required
   relationships.0.type -> Input should be 'WORKS_AT', 'OFFERS', 'INTEGRATES_WITH', 'PARTNERS_WITH' or 'LOCATED_IN'

Structured outputs enforce shape, nothing more. They do not establish that a person works at the seed company or that a named tool is an integration. The extraction prompt supplies those rules: use the seed company's name, exclude customer testimonials, extract named tools rather than counts such as "500+ integrations", and copy supporting text without paraphrasing. The model receives at most 40,000 characters of each page, which bounds the input but can lose entries near the end of a long directory; chunking with overlap and a deduplication pass would extend coverage, and this implementation truncates.

Check quotes before creating relationships

After schema validation, the quote guard checks that each supporting quote occurs in the captured Markdown. It normalises Unicode, Markdown markup and whitespace, then drops entities whose quote is missing. The excerpt below is the entity-filtering part of the walkthrough guard; the full function goes on to check employers and relationship endpoints:

def guard(result, page: str, company: str):
    """Keep entities whose quote is on the page and relationships whose endpoints survived."""
    page_text = normalize(page)
    found = lambda quote: len(quote) > 2 and normalize(quote).rstrip(".,;:!?") in page_text  # noqa: E731
    dropped = []
    for field in ("companies", "people", "products", "integrations", "locations"):
        items = getattr(result, field)
        dropped += [f"{field}: {e.name}" for e in items if not found(e.source_quote)]
        setattr(result, field, [e for e in items if found(e.source_quote)])

Employer filtering removes people the model attached to a clearly different company; people with no employer value stay. The packaged extraction module additionally checks endpoint labels and standardises integration names such as "Stripe for Pipedrive". The recorded output shows both useful rejection and lost recall:

https://basecamp.com/integrations: 19 relationships, 4 dropped
   dropped integrations: Trello
   dropped locations: Chicago
   dropped relationship: Basecamp INTEGRATES_WITH Trello
   dropped relationship: Basecamp LOCATED_IN Chicago
https://www.pipedrive.com/en/partner: 1 relationships, 1 dropped
   dropped relationship: Pipedrive PARTNERS_WITH Zimple

Three different failures hide in those lines. The Chicago quote was absent from the page. Trello was on the page, but the model paraphrased its quote, so the match failed. The Zimple relationship lost its company endpoint before the check ran. All three stop a relationship; only one of them was a hallucination.

That is the boundary to keep in mind. A quote match proves the words were on the page; it does not prove the interpretation, and an unrelated sentence can pass while supporting the wrong entity, title or country. The pipeline also adds relationships the model did not state: retained people get WORKS_AT the seed company, and the company gets OFFERS, INTEGRATES_WITH and LOCATED_IN edges to the other retained entities, each reusing the entity's quote. Those are inferred links, and they can recreate a relationship whose own quote just failed. For anything consequential, verify the full subject-predicate-object claim and record inferred edges separately.

Normalise names, then load with MERGE

Even grounded extractions name the same tool in different ways. Before loading, the normalisation step strips a few domain and legal suffixes, lowercases and removes punctuation, then lets RapidFuzz merge names that score 90 or more, with seed company names taking precedence as the representative:

def key(name: str) -> str:
    """"Stripe, Inc." and "stripe.com" both become "stripe"."""
    name = re.sub(r"\.(com|io|ai|co|app)$", "", name.lower().strip())
    return " ".join(t for t in re.sub(r"[^\w\s&+]", " ", name).split() if t not in SUFFIXES)

'Stripe' vs 'Stripe, Inc.': score 100, merged=True
'Salesforce' vs 'Salesforce.com': score 100, merged=True
'Notion' vs 'Motion': score 83, merged=False

This is approximate name matching, not identity resolution. Two people can share a name, similarly named businesses can be unrelated, and a city needs a country. The packaged implementation matches within a label only, so a Company and an Integration called the same thing stay separate nodes. Company domains, person identifiers and city-country keys would give a larger graph stronger identities.

The loader then uses MERGE so that matching nodes and relationships are reused rather than duplicated:

NODE = """
UNWIND $rows AS row
MERGE (n:{label} {{name: row.name}})
SET n += row.props,
    n.source_urls = CASE WHEN row.url IN coalesce(n.source_urls, []) THEN n.source_urls
                    ELSE coalesce(n.source_urls, []) + row.url END,
    n.rids = CASE WHEN row.rid IN coalesce(n.rids, []) THEN n.rids ELSE coalesce(n.rids, []) + row.rid END
"""

REL = """
UNWIND $rows AS row
MATCH (a:{source} {{name: row.source}}), (b:{target} {{name: row.target}})
MERGE (a)-[r:{type} {{source_url: row.url}}]->(b)
SET r.rid = row.rid, r.extracted_at = row.extracted_at, r.source_quote = row.source_quote
"""

Each label has a uniqueness constraint on name. A relationship's identity is its endpoints, its type and its source_url, so the same connection supported by two pages becomes two edges, each with its own quote, rid and timestamp. Loading the same extracted dataset twice shows the idempotence:

After load 1: 90 nodes, 119 relationships
After load 2: 90 nodes, 119 relationships

Open Neo4j Browser and run MATCH p=(c:Company)-[]->() RETURN p LIMIT 100 to see the result. Repeated loading does not make this a historical or synchronised graph, though. If a page stops mentioning an integration, the old edge stays until something removes it, and a later load of the same edge overwrites its quote, rid and timestamp in place. Versioned observations or a per-page reconciliation step are what keep history or retire stale claims.

Query the connections across company sites

With the graph loaded, three Cypher queries ask which companies share integrations, which people appear on more than one company's site, and which countries hold locations for more than one company:

QUERIES = {
    "Shared integrations": """
        MATCH (a:Company)-[:INTEGRATES_WITH]->(i:Integration)<-[:INTEGRATES_WITH]-(b:Company)
        WHERE a.name < b.name
        RETURN a.name AS company_a, b.name AS company_b,
               count(DISTINCT i) AS shared, collect(DISTINCT i.name)[..6] AS integrations
        ORDER BY shared DESC LIMIT 5""",
    "People on two companies' sites": """
        MATCH (p:Person)-[r:WORKS_AT]->(c:Company)
        WITH p, collect(DISTINCT c.name) AS companies, collect(DISTINCT r.source_url) AS pages
        WHERE size(companies) >= 2
        RETURN p.name AS person, companies, pages""",
    "Geographic clusters": """
        MATCH (c:Company)-[:LOCATED_IN]->(l:Location)
        WITH l.country AS country, collect(DISTINCT c.name) AS companies, collect(DISTINCT l.name) AS cities
        WHERE size(companies) >= 2
        RETURN country, companies, cities ORDER BY size(companies) DESC""",
}

a.name < b.name returns each company pair once, and DISTINCT stops several supporting pages from inflating the integration count. The walkthrough returned:

Shared integrations:
   {"company_a": "Basecamp", "company_b": "Pipedrive", "shared": 1, "integrations": ["Zapier"]}

People on two companies' sites:
   {"person": "Jason Fried", "companies": ["HEY", "Basecamp"], "pages": ["https://www.hey.com/", "https://basecamp.com/", "https://basecamp.com/about"]}

Geographic clusters:
   {"country": "Germany", "companies": ["Pipedrive", "Zendesk"], "cities": ["Berlin"]}
   {"country": "Portugal", "companies": ["Pipedrive", "Zendesk"], "cities": ["Lisbon"]}

Read the rows with the model in mind. Jason Fried appears on both the Basecamp and HEY sites, but both brands belong to 37signals, so two Company nodes do not mean two employers. The geographic query groups by country and collects cities across the matched companies; it does not say that every company has an office in every listed city. The repository's recorded full-pipeline run found four shared integrations for the same pair, Asana, Stripe, Trello and Zapier, and results differ between runs because pages, model outputs and the two implementations differ.

Add GraphRAG for plain-English questions

Cypher makes the graph inspectable; a question is friendlier. The GraphRAG script uses text-to-Cypher: the model receives the schema and the question, writes a query, and then answers from the rows. No embeddings and no vector database are involved. Before anything runs, a keyword check rejects write clauses and multiple statements:

WRITE_WORDS = re.compile(r"\b(CREATE|MERGE|DELETE|DETACH|SET|REMOVE|DROP|LOAD|FOREACH|CALL)\b", re.I)


class Query(BaseModel):
    cypher: str


def check_read_only(cypher: str) -> str:
    """Reject write clauses. String literals are blanked first so names like 'Merge Labs' pass."""
    code = re.sub(r"'[^']*'|\"[^\"]*\"", "''", cypher).strip().rstrip(";")
    if ";" in code or WRITE_WORDS.search(code) or not re.match(r"(?i)\s*(MATCH|OPTIONAL|WITH|UNWIND)", code):
        raise ValueError(f"Blocked query: {cypher}")
    return cypher.strip().rstrip(";")

The query then runs inside execute_read():

with GraphDatabase.driver(NEO4J_URI, auth=NEO4J_AUTH) as driver, driver.session() as session:
    rows = session.execute_read(lambda tx: [r.data() for r in tx.run(cypher)])

Neither line is a security boundary. A regular expression is not a Cypher parser, and the Neo4j driver documentation says plainly that execute_read() does not necessarily imply access control. A deployed question service should enforce read-only access at the database or deployment boundary, and the tutorial's default neo4j credentials, which also load the graph, are not a restricted query identity.

The recorded question shows the retrieval and citation path end to end:

Safety check works: Blocked query: MATCH (n) DETACH DELETE n

Cypher:
MATCH (a:Company)-[r1:INTEGRATES_WITH]->(i)<-[r2:INTEGRATES_WITH]-(b:Company)
WHERE toLower(a.name) = toLower('Basecamp') AND toLower(b.name) = toLower('Pipedrive')
UNWIND [r1, r2] AS r
RETURN i.name AS integration, startNode(r).name AS company, r.source_url AS source_url, r.rid AS rid LIMIT 25

Rows: 2

Answer:
Basecamp and Pipedrive have the integration "Zapier" in common. This is supported by the following sources: [https://basecamp.com/integrations, rid: 86c17c449811ab0e0f861dd3] and [https://www.pipedrive.com/en/marketplace, rid: d2ef101d7ac8d30bf3a26a5a].

The query returns one provenance row per company's connection to Zapier, so the answer can cite both pages and both rids. Those references make the evidence inspectable while the stored objects exist; they do not verify the model's wording claim by claim. The packaged module adds broader keyword filtering, up to three generation attempts, and a 50-row slice applied in Python after the results are materialised, so database-side limits and timeouts still matter on a large graph.

Re-extract stored pages without another crawl

The schema and prompt will change. You might add investors, split products from the companies that make them, or tighten relationship checks, and you want to evaluate each change against the same captured pages rather than a new snapshot of the web. The refresh script retrieves a stored response by rid from Cloud Storage:

def get_stored(rid: str) -> str:
    response = httpx.get(STORAGE, params={"token": CRAWLBASE_TOKEN, "rid": rid}, timeout=60)
    response.raise_for_status()
    print(f"  rid={rid} stored_at={response.headers.get('stored_at')} url={response.headers.get('url')}")
    return response.text

For the recorded format=md request, the retrieved body matched the original Markdown exactly, and a stored generic-extractor response came back as the parsed JSON it had been, not raw HTML. Store and retrieve the representation your extraction consumes rather than assuming every stored object is a page. The walkthrough selects up to three pages, asserts that each stored body matches the saved crawl, and extracts again:

rid=abac8926164450fab0456a27 stored_at=2026-10-29T10:16:17+01:00 url=https://basecamp.com/about
re-extracted 2 relationships, dropped 0
rid=86c17c449811ab0e0f861dd3 stored_at=2026-10-29T10:16:33+01:00 url=https://basecamp.com/integrations
re-extracted 20 relationships, dropped 2

Re-extraction changes the interpretation of a snapshot; it does not fetch current content. A live update is a fresh crawl, then extraction, then load. Stored pages are kept for 30 days; storing one costs half a credit and retrieving it costs nothing. Keep your own snapshots where long-term reproducibility matters, and check the Cloud Storage retention policy for the current terms.

Run the complete pipeline

The packaged CLI joins the nine transformations into commands. From final/, install its dependencies and the package, then run the four-company seed list:

python -m pip install -r requirements.txt
python -m pip install -e .
python -m kg_pipeline run-all --seeds data/seed_small.csv

The recorded full-pipeline run reported:

Fetched 14 pages. Saved to .../final/output/pages.json
Extracted 14 pages: kept 223 items, dropped 11 by the quote guard.
Merged 0 near-duplicate names.
Graph now has 130 nodes and 159 relationships.

A second run against the same database kept those counts. This is a separate run from the walkthrough's 90-node, 119-relationship graph, and the kept figure is counted at the quote guard, before cleanup and inferred links, so it is not the final graph size. One note on provenance: the repository's fixtures reproduce the discovery output and the 223 kept items exactly; the other console output quoted in this post is the authors' recorded run and is not committed as a fixture. The CLI also exposes crawl, extract, load, query, ask and refresh:

python -m kg_pipeline ask "Which integrations do Basecamp and Pipedrive have in common?"
python -m kg_pipeline refresh
python -m kg_pipeline load --seeds data/seed_small.csv

Ordinary crawl and extraction runs reuse cached results: discovery caches by token mode and URL, Markdown by URL, extraction by model, prompt version, company and page-content hash, and nothing expires on its own. Clear the crawl cache before a scheduled recrawl or after changing rendering settings. refresh always makes fresh extraction calls and ask is never cached, so both cost model usage every time. The client uses a 90-second timeout, up to three attempts with exponential backoff for transient statuses, and at least one second between request starts: sequential pacing, not a statement about your account's concurrency.

The test suite holds 112 offline tests and 16 live integration tests. The offline ones cover discovery, schema validation, quote checks, normalisation, query filtering and mocked crawl and cache behaviour on recorded fixtures; the live ones need credentials and a Neo4j, and some clear the database. From final/:

python -m pytest -m "not integration"

The repository's final/tests/fixtures/FINDINGS.md keeps the underlying Crawlbase observations: response shapes, the readability comparison, the token comparison, storage behaviour and the redirect case. data/seed_companies.csv lists 50 seed sites for extending the experiment.

Conclusion

A useful knowledge graph connects every extracted claim to the content that produced it. This pipeline uses the Crawling API to discover and capture company pages, structured LLM extraction to propose entities and relationships, and Neo4j to query their connections, with a page URL, a quote and a stored-page rid on every edge.

Its limits are explicit by design. Quote presence is weaker than semantic verification, name matching is weaker than identity resolution, and idempotent loading is not the same as retiring stale facts. Each of those is a place to invest if the graph grows from a four-company experiment into a research tool. Clone the repository, create a Crawlbase account for the acquisition stage, and start with a seed list small enough that you can read every relationship it produces.

Frequently Asked Questions (FAQs)

When should I use a knowledge graph instead of relational tables?

Both can store these relationships. A graph pays off when your questions repeatedly traverse connections between companies, people, integrations and locations. Fixed-schema reporting and aggregation are still easier in tables.

Does every company website need the JavaScript token?

No. Several recorded pages exposed their content with the Normal token, and the JavaScript token costs twice as much per request. Use it when a page builds its content in the browser, inspect the content rather than trusting a length threshold, and clear the Markdown cache when you change the setting for a seed.

Does a matching source quote guarantee an accurate relationship?

No. It confirms that the normalised quote occurs on the captured page. The quote can still be unrelated to the claim, and the pipeline adds some relationships by inference. Stronger verification checks the entity identities, the relationship meaning and the surrounding context together.

Can I improve extraction without crawling the websites again?

Yes. Retrieve the stored Markdown by rid and re-extract it with the revised prompt or schema while the capture is still within its retention window. In the packaged pipeline that is refresh followed by load. It reprocesses existing evidence; current information needs a fresh crawl.

NZ

Neil Zamora

Senior Architect · Crawlbase

Senior architect at Crawlbase, focused on the systems behind large-scale crawling: proxy rotation, anti-bot resilience, and the APIs that hide that complexity.

IK

Ian Kalvin

Technical Support Engineer · Crawlbase

Technical support engineer at Crawlbase, writing from the front line of what actually breaks in production scraping and proxy setups.