Crawling API / GitHub

GitHub Scraper.
Any page, fully rendered.

Send any GitHub URL and get the fully rendered HTML back, through residential proxies with anti-bot handling built in.
Or pull structured JSON with the github-repository, github-serp and github-profile scrapers.

Start freeSee it live
99% success rate140M residential IPs30 geographies
GitHub URLHTML or JSONgithub.com/anthropics/anthropic-sdk-pythonCrawlbaseRouteRenderExtractRendered HTMLStructured JSONcrawling-apigeneric-extractorgithub.com · rate limit rerouted · 200
Live crawl feed · GitHub1.24M req/minStreaming
200github.com/anthropics/anthropic-sdk-pythonES128ms
200github.com/anthropics/anthropic-sdk-pythonBR218ms
200github.com/anthropics/anthropic-sdk-pythonBR116ms
200github.com/search?q=llm+agents&type=repositoriesBR181ms
301github.com/kubernetes/kubernetes/issuesES109ms
200github.com/search?q=llm+agents&type=repositoriesUS150ms
200github.com/torvalds/linuxJP169ms
200github.com/search?q=web+scraping&type=repositoriesGB189ms
200github.com/facebook/reactJP157ms
200github.com/torvaldsFR44ms
301github.com/openai/openai-pythonUS79ms
200github.com/microsoftES54ms
200github.com/anthropics/anthropic-sdk-pythonCA144ms
200github.com/huggingfaceAU164ms
200github.com/huggingfaceBR198ms
200github.com/topics/machine-learningES86ms
200github.com/torvaldsGB48ms
200github.com/topics/machine-learningUS158ms
200github.com/torvaldsJP80ms
200github.com/facebook/reactFR160ms
200github.com/torvaldsFR208ms
301github.com/vercel/next.js/issuesUS130ms
200github.com/anthropics/anthropic-sdk-pythonJP94ms
200github.com/golang/goGB102ms
200github.com/golang/goUS189ms
200github.com/anthropics/anthropic-sdk-python/issuesUS52ms
200github.com/anthropics/anthropic-sdk-pythonES128ms
200github.com/anthropics/anthropic-sdk-pythonBR218ms
200github.com/anthropics/anthropic-sdk-pythonBR116ms
200github.com/search?q=llm+agents&type=repositoriesBR181ms
301github.com/kubernetes/kubernetes/issuesES109ms
200github.com/search?q=llm+agents&type=repositoriesUS150ms
200github.com/torvalds/linuxJP169ms
200github.com/search?q=web+scraping&type=repositoriesGB189ms
200github.com/facebook/reactJP157ms
200github.com/torvaldsFR44ms
301github.com/openai/openai-pythonUS79ms
200github.com/microsoftES54ms
200github.com/anthropics/anthropic-sdk-pythonCA144ms
200github.com/huggingfaceAU164ms
200github.com/huggingfaceBR198ms
200github.com/topics/machine-learningES86ms
200github.com/torvaldsGB48ms
200github.com/topics/machine-learningUS158ms
200github.com/torvaldsJP80ms
200github.com/facebook/reactFR160ms
200github.com/torvaldsFR208ms
301github.com/vercel/next.js/issuesUS130ms
200github.com/anthropics/anthropic-sdk-pythonJP94ms
200github.com/golang/goGB102ms
200github.com/golang/goUS189ms
200github.com/anthropics/anthropic-sdk-python/issuesUS52ms
01 Live demo

Three GitHub scrapers, live.

The Crawling API, typed live. Watch the github-repository, github-serp and github-profile scrapers each return structured JSON. Hover to pause and read.

ready
keys 1-3 switch · click to pauserun your own URL
Run your first request in minutes. Up to 20,000 free requests, no credit card.Start free
02 Capabilities

One API, everything GitHub throws at you.

GitHub renders repos client-side and rate limits aggressively. The Crawling API loads the full page in a real browser, reaches it through residential IPs, and hands you clean HTML or JSON.

render

Full JavaScript rendering

A real browser executes the page, so the README, stars, forks, languages and file tree all render and land in the HTML, not just the initial markup.

proxies

140M residential IPs

Every request rotates a residential IP across 30 geographies, so you reach GitHub like a real local visitor.

anti-bot

Blocks handled for you

Rate limits, bot checks and challenge pages are cleared automatically. No personal access tokens to juggle, nothing to maintain.

format

HTML or JSON

Get the full rendered HTML, or add scraper=github-repository, github-serp or github-profile to return repositories, search results and profiles as structured JSON.

extras

Screenshots and async

The same call can capture a full-page screenshot, or run asynchronously with webhooks and cloud storage.

one token

One API for every site

The Crawling API works on any URL, so the same token covers GitHub and everything else you crawl. See the live demo.

03 Output

Every repository field, as clean JSON.

Send a repository URL with scraper=github-repository and the repository comes back as typed JSON. Swap in github-serp or github-profile to do the same for search results and profiles.

{ "name": "rails", "fullName": "rails/rails", "url": "https://github.com/rails/rails", "description": "Ruby on Rails", "stars": 58789, "forks": 22414, "watchers": 3819, "primaryLanguage": "Ruby", "languages": [ "Ruby", "JavaScript", "HTML" ], "topics": [ "ruby", "rails", "framework" ], "openIssues": 612, "latestRelease": "v8.1.3", "defaultBranch": "main", "license": "MIT license", "archived": false }

Identity

name · string  fullName · string  url · string  description · string

Popularity

stars · number  forks · number  watchers · number

Stack

primaryLanguage · string  languages · array  topics · array

Activity

openIssues · number  latestRelease · string  defaultBranch · string

Meta

license · string  archived · boolean

04 How it works

From URL to data in one call.

Every GitHub request moves through the same path. You send a URL, we operate everything in between.

01

Send the URL

Pass any public GitHub URL with your token: a repository, an issue, a profile or a search.

02

Rotate a proxy

A residential IP and geography that reach GitHub cleanly, drawn from 140M IPs across 30 regions.

03

Render the page

A real browser loads the repository page so the README, stars, forks, languages and file tree render before capture.

04

Clear anti-bot

GitHub's rate limits and bot checks are handled automatically. No tokens to juggle, nothing to maintain.

05

Return HTML or JSON

The fully rendered HTML comes back, or typed JSON when you add a scraper like github-repository, github-serp or github-profile.

05 Use cases

What teams build on GitHub data.

USE / 01OSS

Open-source & developer intelligence

Track repositories, stars, forks, languages and releases to map the open-source landscape and who is building what.

USE / 02Trends

Dependency & trend tracking

Watch topics, search pages and release activity to spot rising libraries, frameworks and tooling early.

USE / 03Recruiting

Recruiting & dev-rel

Mine public profiles and contributions to find maintainers, source talent and reach the right developer communities.

USE / 04Training

Training data for code models

Feed clean README and page text into models, RAG pipelines and coding agents through one API.

USE / 05Security

Security & supply-chain research

Monitor public repositories, issues and dependencies to study packages and supply-chain risk.

USE / 06Coverage

Any URL, one API

Crawl repos, issues, profiles, topics and search, plus any other site you need.

06 Notes

Good to know when scraping GitHub.

Rendered like a real browser

GitHub renders repos client-side; the Crawling API runs a real browser so the README, stars, forks, languages and file tree all load before capture.

HTML by default, JSON on request

You get the full rendered HTML. Add scraper=github-repository, github-serp or github-profile for parsed JSON, or parse the HTML yourself.

No GitHub tokens or API limits

You point at the public web page, so there are no personal access tokens to rotate and no REST API rate limits to budget around.

Public repos and profiles only

The Crawling API reads publicly visible pages, with no login, so you get what a logged-out visitor sees.

07 Why Crawlbase

Built to crawl GitHub at scale.

The Crawling API runs on the same network that serves 46,000+ paying customers and 70,000+ developers. No proxies to buy, no browsers to run, nothing to patch when GitHub changes.

99%
Average request success rate
140M
Residential IPs, plus 98M datacenter
30
Geographies for accurate local results
20/s
Requests per second by default, more on request

One token, official SDKs for Python, Node and Ruby, and a 99.99% uptime network underneath.

08 FAQ

GitHub scraping questions.

Send the GitHub URL to the Crawlbase Crawling API with your token. Crawlbase rotates a residential proxy, renders the page in a real browser, clears bot checks, and returns the fully rendered HTML. Add a dedicated scraper (scraper=github-repository, github-serp or github-profile) to get structured JSON instead.
Yes. By default the Crawling API returns rendered HTML; add a dedicated GitHub scraper (scraper=github-repository for repos, github-serp for search, github-profile for profiles) to receive structured JSON, or parse the HTML yourself.
No. You point the Crawling API at the public web page, so there are no personal access tokens to issue or rotate and no REST API rate limits to budget around. Crawlbase reaches GitHub for you.
Crawlbase routes each request through rotating residential IPs across 30 geographies and clears bot checks automatically. You do not manage proxies or juggle tokens, and there is nothing to maintain when GitHub changes its setup.
Yes. A real browser loads the repository page, so the rendered README, stars, forks, languages and file tree are all present in the HTML, not just the initial markup.
Any public URL: repositories, issues and pull requests, user and organization profiles, topic pages and search results. The same API works on any other site too.
Start free with up to 20,000 requests and no credit card. Paid plans scale with usage, and the same token works across the Crawling API and every Crawlbase scraper.

Start scraping GitHub.
Skip the tokens and rate limits.

Free to begin with up to 20,000 requests. One token for the Crawling API and every scraper.

Start freeRead the docs