Developer

Overview

The Developer category covers the platforms engineering teams pull from for open-source intelligence, dependency and ecosystem analysis, developer-relations research, hiring/sourcing signals, and technical question-and-answer mining. Every scraper accepts a target URL, returns parsed JSON in milliseconds, and rides the same residential proxies and anti-bot bypass that powers the Crawling API - same uptime SLA, same one-token authentication, no per-target setup.

The core set targets GitHub, the largest host of public source code. Pick a scraper by the surface you need: a single repository page, a repository search results page (SERP), or a user/organization profile. Each scraper targets a single page-type so the JSON shape stays stable as the underlying HTML changes - and you only pay for successful responses, so retries against a flaky upstream don't show up on your bill.

Stack Overflow is covered too. It is part of the Stack Exchange network, so the same two host-agnostic scrapers handle it - there is no separate Stack Overflow scraper to learn. Point stackexchange-serp at a question list, tag, or search page (e.g. https://stackoverflow.com/questions/tagged/python) for a structured array of questions with scores, answer and view counts, and tags; point stackexchange-thread at a single question (e.g. https://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-processing-an-unsorted-array) for the full body plus every answer and comment. The network parser handles the markup whether the URL is on stackoverflow.com or any other *.stackexchange.com site.

Exercism is covered by four scrapers for the coding-practice platform. Point exercism-serp at a track exercise list (e.g. https://exercism.org/tracks/ruby/exercises) for every exercise with its slug, difficulty, and blurb; exercism-exercise at a single exercise (e.g. /tracks/ruby/exercises/two-fer) for its full instructions as text and HTML; exercism-solutions at an exercise's community solutions for a paginated array of published solutions with author, language, and star counts; and exercism-solution at a single published solution (e.g. /tracks/ruby/exercises/two-fer/solutions/handle) for its iterations, metadata, and source code.

Common pipelines:

• Ecosystem monitoring: poll github-repository on the projects you depend on, snapshot stars, forks, openIssuesCount, latestRelease, and archived, and alert on Δ.
• Discovery: feed github-serp queries (e.g. a topic or keyword search) into a list of github-repository calls to enrich each hit with full metadata.
• Developer sourcing / DevRel: resolve a github-profile to read followers, publicRepos, pinnedRepos, and organizations.
• Dependency intelligence: track primaryLanguage, languages, license, and topics across a portfolio of repositories to flag license or maintenance risk.
• Q&A mining: run stackexchange-serp on a tag or search (e.g. stackoverflow.com/questions/tagged/python), then fan out to stackexchange-thread per question to capture accepted answers, code blocks, and vote scores - a clean corpus for support automation or model training.
• Solution mining: run exercism-serp on a track, fan out to exercism-solutions per exercise, then exercism-solution per author to build a labelled corpus of working code across languages for the same problem.

Every field maps directly to what the page renders, and nullable fields come back as null when the source page omits the value rather than silently disappearing - so your schema stays predictable across calls. No GitHub or Stack Exchange API token, rate-limit juggling, or pagination bookkeeping on your end: the scraper handles the fetch and the parse, and you get back the fields you actually need.

GitHub

Three scrapers covering the GitHub surfaces teams query most - a single repository page, repository search results, and user or organization profiles.

• GitHub Repository - repository page (description, language, stars, forks, issues, license, latest release).
• GitHub SERP - repository search-results page on GitHub.
• GitHub Profile - user or organization profile (bio, followers, pinned repos, organizations).

Stack Overflow

Stack Overflow is served by the Stack Exchange scrapers - point them at stackoverflow.com URLs. Use stackexchange-serp for question lists, tag pages, and search results, and stackexchange-thread for a single question with its full answer thread. Both are host-agnostic across the Stack Exchange network, so the same call shape works for every *.stackexchange.com site.

• Stack Overflow Questions - a Stack Overflow questions, tagged (e.g. /questions/tagged/python), or search-results page as a structured array with scores, answer and view counts, tags, and pagination.
• Stack Overflow Thread - a single Stack Overflow question with its full body plus every answer and comment, with scores, accepted state, and authors.

Exercism

Four scrapers covering the Exercism coding-practice platform - a track exercise listing, a single exercise with its instructions, an exercise community-solutions listing, and a single published solution. Point them at exercism.org URLs; track and exercise slugs are read from the URL path.

• Exercism Exercises - a track exercise-list page (e.g. /tracks/ruby/exercises) as a structured array of exercises with slug, title, difficulty, and blurb.
• Exercism Exercise - a single exercise overview page with its title, difficulty, and full instructions as text and HTML.
• Exercism Solutions - an exercise community-solutions page as a paginated array of published solutions with author, language, stars, and iteration counts.
• Exercism Solution - a single published community solution with its iterations, language, star count, and source code.

Kaggle

Four scrapers covering the Kaggle data-science platform - dataset search, a single dataset with its files and license, notebook search, and a single notebook with its metadata and inputs. Point them at kaggle.com URLs; owner and dataset or notebook slugs are read from the URL path.

• Kaggle Dataset Search - a dataset search or listing page (e.g. /datasets?search=heart+disease) as a ranked array of datasets with owner, size, usability rating, downloads, and notebook count.
• Kaggle Dataset - a single dataset page with its description, keywords, owner, license, file list, and engagement counts.
• Kaggle Notebook Search - a notebook search or listing page (e.g. /code?searchQuery=titanic) as a ranked array of notebooks with author, co-authors, competition context, votes, and comments.
• Kaggle Notebook - a single notebook page with its author, language, runtime, version history, engagement counts, and attached inputs.

Example call

Below: a single github-repository call. Replace YOUR_TOKEN with your Crawling API token; the only required parameters are the target URL and the scraper name.

curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \
  --data-urlencode 'url=https://github.com/rails/rails' \
  --data-urlencode 'scraper=github-repository' -G

Sample response

{
  "name": "rails",
  "owner": "rails",
  "fullName": "rails/rails",
  "url": "https://github.com/rails/rails",
  "description": "Ruby on Rails",
  "primaryLanguage": "Ruby",
  "languages": ["Ruby", "JavaScript", "HTML", "SCSS", "CSS", "Dockerfile"],
  "stars": 58789,
  "forks": 22414,
  "watchers": 2400,
  "hasIssues": true,
  "openIssuesCount": 478,
  "openPrsCount": 1081,
  "topics": ["ruby", "rails", "html", "activerecord", "framework", "mvc", "activejob"],
  "license": "MIT license",
  "defaultBranch": "main",
  "latestRelease": "v8.1.3",
  "readmePresent": true,
  "archived": false
}

Full reference (parameters, all 4 SDK languages, edge cases): GitHub Repository - full reference