Log in

GitHub SERP

Parse a GitHub repository search results page into a structured array of repositories with stars, language, topics, and pagination info.

API usage

Add &scraper=github-serp to a Crawling API request. URL-encode the target URL in the url parameter. One request returns one page; follow pagination.next_page_url to walk further, up to GitHub's 1000-result cap.

curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \
  --data-urlencode 'url=https://github.com/search?q=web+scraping&type=repositories' \
  --data-urlencode 'scraper=github-serp' -G
from crawlbase import CrawlingAPI

api = CrawlingAPI({'token': 'YOUR_TOKEN'})
res = api.get(
    'https://github.com/search?q=web+scraping&type=repositories',
    {'scraper': 'github-serp'}
)

import json
data = json.loads(res['body'])
const { CrawlingAPI } = require('crawlbase');
const api = new CrawlingAPI({ token: 'YOUR_TOKEN' });

const res = await api.get(
  'https://github.com/search?q=web+scraping&type=repositories',
  { scraper: 'github-serp' }
);
const data = JSON.parse(res.body);
require 'crawlbase'
api = Crawlbase::API.new(token: 'YOUR_TOKEN')

res = api.get('https://github.com/search?q=web+scraping&type=repositories', scraper: 'github-serp')
data = JSON.parse(res.body)

Example input URL

The URL passed in the url parameter (URL-decoded for readability):

https://github.com/search?q=web+scraping&type=repositories

Response shape

JSON response body. Field types may be null when the source page omits the value.

query
string | null
Search query that produced these results.
searchType
string | null
Search type, e.g. repositories.
sort
string | null
Sort field, e.g. stars. Absent when sorting by best match.
order
string | null
Sort direction, desc or asc.
totalResults
integer | null
Reported total result count across all pages.
currentPage
integer
Current page number.
pagination
object
Pagination info for walking the result set.
pagination.current_page
integer
Current page number.
pagination.total_pages
integer | null
Highest page number GitHub exposes for this query.
pagination.next_page_url
string | null
Absolute URL of the next page, or null at the last page / result cap.
pagination.previous_page_url
string | null
Absolute URL of the previous page, or null on the first page.
pagination.has_next
boolean
True while a next page exists below the 1000-result cap.
results
array
Per-repository results (see fields below).
results[].position
integer
Rank of the result on this page (1-based).
results[].name
string | null
Repository name.
results[].owner
string | null
Owner (user or organization) login.
results[].fullName
string
Full owner/name identifier.
results[].url
string
Absolute repository URL.
results[].description
string | null
Repository description.
results[].primaryLanguage
string | null
Primary programming language.
results[].stars
integer | null
Star count.
results[].topics
array
Topic labels shown on the result card.
results[].updatedAt
string | null
Last-updated timestamp or relative label, when present.
results[].license
string | null
License shown on the result card, when present.
results[].archived
boolean
True if the repository is a public archive.
results[].sponsorable
boolean
True if the owner can be sponsored.

Sample response

{
  "query": "web scraping",
  "searchType": "repositories",
  "sort": "stars",
  "order": "desc",
  "totalResults": 133816,
  "currentPage": 1,
  "pagination": {
    "current_page": 1,
    "total_pages": 100,
    "next_page_url": "https://github.com/search?q=web+scraping&type=repositories&s=stars&o=desc&p=2",
    "previous_page_url": null,
    "has_next": true
  },
  "results": [
    {
      "position": 1,
      "name": "scrapy",
      "owner": "scrapy",
      "fullName": "scrapy/scrapy",
      "url": "https://github.com/scrapy/scrapy",
      "description": "Scrapy, a fast high-level web crawling & scraping framework for Python.",
      "primaryLanguage": "Python",
      "stars": 63119,
      "topics": ["python", "crawler", "framework", "scraping", "crawling"],
      "updatedAt": null,
      "license": null,
      "archived": false,
      "sponsorable": false
    }
  ]
}