Crunchbase is one of the largest directories of company and startup data on the web. Public company profiles carry the kind of structured intelligence that powers market research, competitor analysis, sales prospecting, and funding research: a company name, what it does, where it is based, headcount, its website, its rank, and who founded it. The catch is that Crunchbase renders those profiles client-side and challenges automated traffic hard, so a plain HTTP request hands you an empty shell instead of the data you came for.
This guide shows you how to scrape Crunchbase with Python the reliable way. You build a small, runnable scraper that fetches a rendered company profile through the Crawling API, parses the fields you want with BeautifulSoup, and prints clean structured output. We keep the whole walkthrough scoped to public profile data, and the legality section near the end is not boilerplate, so read it before you point this at any real volume.
What you will build
A Python script that takes a public Crunchbase company URL, retrieves the rendered HTML through the Crawling API, and extracts a structured record of the company. We will target a public organization profile as the running example and pull these fields:
- Title the company name, for example "OpenAI".
- Description a short summary of what the company does.
- Location where the company is based, like "San Francisco, California".
- Employees the headcount band, for instance "1001-5000".
- Company URL the link to the company's own website.
- Rank the company's position in Crunchbase's ranking.
- Founded the year the company was established.
- Founders the people who started the company.
Why a plain fetch fails on Crunchbase
If you request a Crunchbase organization URL with a bare HTTP client, you get a response with status 200 and almost none of the profile data in the body. Two things work against you. First, Crunchbase renders its profile content in the browser with JavaScript, so the initial HTML is a shell that only fills in after the page's scripts run. Second, Crunchbase flags automated traffic quickly: datacenter IPs and request patterns that do not look like a real browser get challenged or blocked before they ever reach the rendered content.
So a working Crunchbase scraper needs two things in one request: a browser that actually renders the page, and an IP the platform reads as a real visitor. You can assemble that yourself with a headless browser plus a pool of rotating residential proxies, but stitching those together and keeping them healthy is most of the work. The Crawling API folds both into a single call: you send it the URL with a JavaScript token, it renders the page behind a trusted IP, and it returns finished HTML for you to parse.
Crawlbase offers two token types. The normal token fetches static HTML; the JavaScript (JS) token renders the page in a real browser first. Crunchbase is client-side rendered, so you need the JS token here. Using the normal token returns the same empty shell a plain fetch would, and there is nothing to parse out of it.
Prerequisites
You need a few things in place before writing any code. None of them take long.
Basic Python. You should be comfortable writing and running a Python script and installing packages with pip. If you are new to the language, the official Python docs and any beginner course will get you to the level this tutorial assumes.
Python 3.8 or later. Confirm your version with python --version. If you do not have it, install it from python.org or through a distribution like Anaconda.
A Crawlbase account and JS token. Sign up, open your dashboard, and copy your JavaScript (JS) token from the account docs page. Treat the token like a password: it authenticates your requests, so keep it out of version control.
Set up the project
Create a virtual environment so project dependencies stay isolated, then install the two libraries the scraper needs.
python --version python -m venv crunchbase_env source crunchbase_env/bin/activate pip install crawlbase beautifulsoup4
On Windows, activate the environment with crunchbase_env\Scripts\activate instead of the source line. Two dependencies do the work: crawlbase is the official client for the Crawling API, and beautifulsoup4 parses the returned HTML so you can pull out individual fields by CSS selector.
Step 1: Fetch the rendered profile
Start by getting the finished page. Import the CrawlingAPI class, initialize it with your JS token, and request the company URL. Checking the status code before you parse keeps failures loud instead of silent.
from crawlbase import CrawlingAPI api = CrawlingAPI({"token": "YOUR_CRAWLBASE_JS_TOKEN"}) def crawl(page_url): options = {"ajax_wait": "true", "page_wait": 5000} response = api.get(page_url, options) if response["status_code"] == 200: return response["body"].decode("utf-8") print(f"Request failed: {response['status_code']}") return None if __name__ == "__main__": page_url = "https://www.crunchbase.com/organization/openai" html = crawl(page_url) print(html[:500] if html else "No HTML returned")
The two wait options matter for a client-rendered target like this. ajax_wait tells the API to wait for asynchronous content to finish loading, and page_wait holds for a fixed number of milliseconds after load so late-rendering elements appear before the page is captured. Five seconds is a reasonable start; raise it if the profile fields come back empty. Run the script with python scraper.py and you should see real profile markup, not the empty shell a plain fetch returns. That confirms rendering works before you write a single selector.
Crunchbase needs a rendered page behind a trusted IP, in one call. The Crawling API takes a JS token, runs the page in a real browser, rotates through residential IPs server-side, and hands you finished HTML, so you skip running a headless fleet and a proxy pool yourself. Point it at a public company profile on the free tier first.
Step 2: Parse the company fields with BeautifulSoup
With rendered HTML in hand, load it into BeautifulSoup and pull each field by its selector. Crunchbase profiles lay the core details out in a predictable structure, so you can map title, description, location, headcount, website, rank, founding year, and founders to individual selectors. Wrap the whole extraction in a try/except so one missing field does not crash the run.
from bs4 import BeautifulSoup def text_of(soup, selector): el = soup.select_one(selector) return el.get_text(strip=True) if el else None def scrape_company(html): soup = BeautifulSoup(html, "html.parser") rows = ".section-content-wrapper li.ng-star-inserted" company_link = soup.select_one(f"{rows}:nth-of-type(5) a[role='link']") return { "title": text_of(soup, "h1.profile-name"), "description": text_of(soup, "span.description"), "location": text_of(soup, f"{rows}:nth-of-type(1)"), "employees": text_of(soup, f"{rows}:nth-of-type(2)"), "company_url": company_link["href"] if company_link else None, "rank": text_of(soup, f"{rows}:nth-of-type(6) span"), "founded": text_of(soup, ".text_and_value li:nth-of-type(4) field-formatter"), "founders": text_of(soup, ".text_and_value li:nth-of-type(5) field-formatter"), }
The text_of helper does two useful things at once: it queries a single element and returns None when the element is missing, instead of throwing on a .get_text() call against nothing. That keeps the extraction resilient when one field is absent on a given profile, which is common since not every company lists a rank or a website. The company URL is read from an anchor's href rather than its text, so it is handled separately.
Crunchbase's class names (the Angular ng-star-inserted markers, the field-formatter elements, and the section wrappers) change without notice. Treat the selectors above as a starting template, not a contract. When a field comes back as None, re-inspect the live profile in your browser's dev tools and update the selector. Periodic selector maintenance is normal for any production scraper, not a sign something is broken.
Step 3: Put it together
Now wire the fetch and the parse into one runnable script. Fetch the rendered HTML, hand it to the parser, and print the structured record.
import json from crawlbase import CrawlingAPI from bs4 import BeautifulSoup api = CrawlingAPI({"token": "YOUR_CRAWLBASE_JS_TOKEN"}) def crawl(page_url): options = {"ajax_wait": "true", "page_wait": 5000} response = api.get(page_url, options) if response["status_code"] == 200: return response["body"].decode("utf-8") print(f"Request failed: {response['status_code']}") return None def text_of(soup, selector): el = soup.select_one(selector) return el.get_text(strip=True) if el else None def scrape_company(html): soup = BeautifulSoup(html, "html.parser") rows = ".section-content-wrapper li.ng-star-inserted" company_link = soup.select_one(f"{rows}:nth-of-type(5) a[role='link']") return { "title": text_of(soup, "h1.profile-name"), "description": text_of(soup, "span.description"), "location": text_of(soup, f"{rows}:nth-of-type(1)"), "employees": text_of(soup, f"{rows}:nth-of-type(2)"), "company_url": company_link["href"] if company_link else None, "rank": text_of(soup, f"{rows}:nth-of-type(6) span"), "founded": text_of(soup, ".text_and_value li:nth-of-type(4) field-formatter"), "founders": text_of(soup, ".text_and_value li:nth-of-type(5) field-formatter"), } def main(): page_url = "https://www.crunchbase.com/organization/openai" html = crawl(page_url) if not html: return data = scrape_company(html) print(json.dumps(data, indent=2)) if __name__ == "__main__": main()
What the output looks like
Run the full script with python scraper.py and you get a clean structured record for the company, ready to write to JSON, CSV, or a database.
{ "title": "OpenAI", "description": "OpenAI is an AI research and deployment company.", "location": "San Francisco, California, United States", "employees": "1001-5000", "company_url": "https://openai.com", "rank": "5", "founded": "2015", "founders": "Elon Musk, Greg Brockman, Ilya Sutskever, Sam Altman" }
Scaling to many companies
One profile is a demo; a real job runs over a list of companies. The shape stays the same: keep a list of organization URLs, fetch each through the Crawling API, parse it with the same function, and collect the rows. Because every profile shares the same structure, the parser you already wrote works across all of them without changes.
companies = [ "https://www.crunchbase.com/organization/openai", "https://www.crunchbase.com/organization/anthropic", ] results = [] for url in companies: html = crawl(url) if html: results.append(scrape_company(html)) with open("companies.json", "w") as f: json.dump(results, f, indent=2)
To find company URLs at scale you can scrape Crunchbase's public search and discover pages with the same fetch-then-parse pattern, collecting the organization links and then visiting each one. Just keep the volume reasonable and respect the rate limits covered below.
Staying unblocked
Even with rendering handled, Crunchbase watches for scraper-shaped traffic. A few habits keep a run healthy, and they apply to any hard commercial target.
- Pace your requests. Hammering profiles in a tight loop is the fastest way to get throttled. Spread requests out and vary your targets instead of crawling one path at full speed.
- Lean on rotation. A pool of residential IPs spreads requests across many real-user addresses so no single one trips a rate limit. The Crawling API handles this for you; if you roll your own stack, this is the part to get right.
- Read the status codes. A run that starts returning challenges or errors is telling you the current rate or IP tier is no longer enough. Treat that as signal to back off, not noise to ignore.
For the broader playbook, see how to scrape websites without getting blocked and the deeper dive on how to bypass captchas while web scraping. If you would rather route your own traffic through a rotating pool instead of using the managed API, the Smart AI Proxy (also called the AI Proxy) gives you the same residential IP rotation as a drop-in proxy endpoint.
Is it legal to scrape Crunchbase?
Whether scraping Crunchbase is allowed depends on Crunchbase's terms of service, your jurisdiction, and what you do with the data. Crunchbase's terms restrict automated access, so scraping can run against those terms regardless of how careful your tooling is. None of the code here changes that; it just makes the technical part work. Read the Crunchbase Terms of Service and its robots.txt, and treat both as the boundary for what you collect.
A few lines worth holding to. Collect only public data: company name, description, location, headcount band, website, rank, founding year, and founders that anyone can see without an account. Respect Crunchbase's stated rate expectations and keep your request volume low enough that you are not straining its servers. Avoid personal data, including anything tied to identifiable individuals beyond what is publicly listed on a company profile. If you plan to reuse the data commercially, get permission or an official agreement rather than assuming silence is consent.
For licensed or bulk access, Crunchbase offers an official Crunchbase API, and that is the right tool when you need large volumes, guaranteed structure, or commercial rights. This guide is deliberately scoped to public company-profile pages because that is the line that keeps the work defensible. It does not cover anything behind a login, Crunchbase Pro or paid-tier data, private or financial data gated by a sign-in, user account or profile data, or any attempt to bypass authentication. If your project needs more than public profiles, the official Crunchbase API or a data agreement is the correct path, not a cleverer scraper.
Key takeaways
- Crunchbase is client-side rendered. A plain fetch returns an empty shell, so you must render the page before you parse it.
-
You need rendering and a trusted IP together. The Crawling API with a JS token does both in one call;
ajax_waitandpage_waitcontrol how long it waits for content. - BeautifulSoup does the extraction. Map title, description, location, employees, website, rank, founded, and founders to current selectors, and expect those selectors to drift.
- Scale by looping URLs. The same parser works across every profile, so a real job is just a list of organization links plus sensible pacing.
- Stay on public data. Respect Crunchbase's ToS and robots.txt, prefer the official Crunchbase API for licensed or bulk data, and never touch accounts, Pro data, or personal information.
Frequently Asked Questions (FAQs)
Why does a plain fetch return no data from Crunchbase?
Because Crunchbase renders its profile content client-side with JavaScript. The initial HTML is a shell that only fills in after the page's scripts run in a browser, so a raw HTTP request returns status 200 with the company fields blank. To get real data you have to render the page first, which is what the Crawling API's JS token handles for you.
Do I need the normal token or the JS token for Crunchbase?
The JS token. The normal token fetches static HTML, which on Crunchbase is the same empty shell a plain fetch returns. The JS token renders the page in a real browser before handing back the HTML, so the profile fields are present when BeautifulSoup parses them.
My selectors return None. What changed?
Almost certainly Crunchbase's markup. Its Angular ng-star-inserted markers, field-formatter elements, and section wrappers change without notice, so selectors that worked last month can break. Re-inspect a live profile in your browser's dev tools and update the selectors. Periodic selector maintenance is normal for any production scraper.
Should I use the official Crunchbase API or scrape the site?
If you need licensed data, bulk volume, guaranteed structure, or commercial reuse rights, use the official Crunchbase API. It is built for that and keeps you on the right side of their terms. Scraping public profiles with the approach in this guide fits smaller, public-data research where no API access is in place, as long as you respect the ToS, robots.txt, and rate limits.
Can I scrape financial or Pro data from Crunchbase?
No, and this guide does not cover it. Financial details and Crunchbase Pro data sit behind a login or a paid tier, so they are not public data. Scraping login-walled or paid content, or bypassing authentication to reach it, is out of scope here and runs against Crunchbase's terms. For that data the correct route is the official Crunchbase API or a licensing agreement.
How do I avoid getting blocked while scraping Crunchbase?
Keep your per-IP request rate low, vary your targets instead of looping one path, and route through rotating residential IPs so no single address trips a rate limit. The Crawling API manages rotation and a trusted IP pool for you; if you build your own stack, that is the part to invest in. Watch the status codes and back off when you start seeing challenges.
Crawl any site at scale, without fighting infrastructure.
Crawlbase handles proxies, fingerprints, and CAPTCHAs so your team ships data pipelines instead of maintaining crawl plumbing. 1,000 requests free, no card required.