Log in

Developer

Scrapers for developer platforms. Point at a GitHub URL and get clean structured JSON - repository metadata, search results, or profiles - instead of HTML.

Overview

The Developer category covers the platforms engineering teams pull from for open-source intelligence, dependency and ecosystem analysis, developer-relations research, and hiring/sourcing signals. Every scraper accepts a target URL, returns parsed JSON in milliseconds, and rides the same residential proxies and anti-bot bypass that powers the Crawling API - same uptime SLA, same one-token authentication, no per-target setup.

Today the category focuses on GitHub, the largest host of public source code. Pick a scraper by the surface you need: a single repository page, a repository search results page (SERP), or a user/organization profile. Each scraper targets a single page-type so the JSON shape stays stable as the underlying HTML changes - and you only pay for successful responses, so retries against a flaky upstream don't show up on your bill.

Common pipelines:

  • Ecosystem monitoring: poll github-repository on the projects you depend on, snapshot stars, forks, openIssuesCount, latestRelease, and archived, and alert on Δ.
  • Discovery: feed github-serp queries (e.g. a topic or keyword search) into a list of github-repository calls to enrich each hit with full metadata.
  • Developer sourcing / DevRel: resolve a github-profile to read followers, publicRepos, pinnedRepos, and organizations.
  • Dependency intelligence: track primaryLanguage, languages, license, and topics across a portfolio of repositories to flag license or maintenance risk.

Every field maps directly to what the page renders, and nullable fields come back as null when the source page omits the value rather than silently disappearing - so your schema stays predictable across calls. No GitHub API token, rate-limit juggling, or pagination bookkeeping on your end: the scraper handles the fetch and the parse, and you get back the fields you actually need.

GitHub

Three scrapers covering the GitHub surfaces teams query most - a single repository page, repository search results, and user or organization profiles.

  • GitHub Repository - repository page (description, language, stars, forks, issues, license, latest release).
  • GitHub SERP - repository search-results page on GitHub.
  • GitHub Profile - user or organization profile (bio, followers, pinned repos, organizations).

Example call

Below: a single github-repository call. Replace YOUR_TOKEN with your Crawling API token; the only required parameters are the target URL and the scraper name.

curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \
  --data-urlencode 'url=https://github.com/rails/rails' \
  --data-urlencode 'scraper=github-repository' -G

Sample response

{
  "name": "rails",
  "owner": "rails",
  "fullName": "rails/rails",
  "url": "https://github.com/rails/rails",
  "description": "Ruby on Rails",
  "primaryLanguage": "Ruby",
  "languages": ["Ruby", "JavaScript", "HTML", "SCSS", "CSS", "Dockerfile"],
  "stars": 58789,
  "forks": 22414,
  "watchers": 2400,
  "hasIssues": true,
  "openIssuesCount": 478,
  "openPrsCount": 1081,
  "topics": ["ruby", "rails", "html", "activerecord", "framework", "mvc", "activejob"],
  "license": "MIT license",
  "defaultBranch": "main",
  "latestRelease": "v8.1.3",
  "readmePresent": true,
  "archived": false
}

Full reference (parameters, all 4 SDK languages, edge cases): GitHub Repository - full reference