crawlbaseDocs
Log in

Stack Exchange Questions

Parse a Stack Exchange questions, tagged, or search results page into structured JSON with each question title, score, answer and view counts, tags, excerpt, author, and pagination.

API usage

Add &scraper=stackexchange-serp to a Crawling API request. URL-encode the target URL in the url parameter.

curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \
  --data-urlencode 'url=https://stackoverflow.com/questions/tagged/web-scraping' \
  --data-urlencode 'scraper=stackexchange-serp' -G
from crawlbase import CrawlingAPI

api = CrawlingAPI({'token': 'YOUR_TOKEN'})
res = api.get(
    'https://stackoverflow.com/questions/tagged/web-scraping',
    {'scraper': 'stackexchange-serp'}
)

import json
data = json.loads(res['body'])
const { CrawlingAPI } = require('crawlbase');
const api = new CrawlingAPI({ token: 'YOUR_TOKEN' });

const res = await api.get(
  'https://stackoverflow.com/questions/tagged/web-scraping',
  { scraper: 'stackexchange-serp' }
);
const data = JSON.parse(res.body);
require 'crawlbase'
api = Crawlbase::API.new(token: 'YOUR_TOKEN')

res = api.get('https://stackoverflow.com/questions/tagged/web-scraping', scraper: 'stackexchange-serp')
data = JSON.parse(res.body)

Example input URL

Any Stack Exchange listing works in the url parameter: a tag, search, or questions page on any site in the network. For example:

https://stackoverflow.com/questions/tagged/web-scraping
https://superuser.com/search?q=ssh+tunnel
https://askubuntu.com/questions?tab=Votes
https://unix.stackexchange.com/questions/tagged/bash
https://mathoverflow.net/questions

Response shape

JSON response body. Field types may be null when the source page omits the value.

site
string
Stack Exchange site the results came from (for example stackoverflow).
query
string
Search query or tag that produced these results.
tag
string | null
Tag the listing was scoped to, or null when not a tag page.
sort
string
Sort order applied to the listing (for example newest, votes, active).
currentPage
integer
Page number of the listing currently returned (1-based).
totalQuestions
integer | null
Total number of questions matching the listing, or null when the page omits the count.
results
array
Questions on this page, in listing order.
results[].position
integer
Position of the question in the returned list (1-based).
results[].id
string
Stack Exchange question id.
results[].title
string
Question title.
results[].questionUrl
string
Canonical URL of the question.
results[].score
integer
Question score (upvotes minus downvotes).
results[].answerCount
integer
Number of answers on the question.
results[].viewCount
integer
Number of views on the question.
results[].hasAcceptedAnswer
boolean
True when the question has an accepted answer.
results[].excerpt
string
Short excerpt of the question body.
results[].tags
array
Tags applied to the question.
results[].author
string
Display name of the question author.
results[].authorUrl
string
Profile URL of the question author.
results[].askedAt
string
Question creation time (ISO 8601).
resultCount
integer
Number of results returned in results.
pagination
object
Pagination cursor for the next page.
pagination.currentPage
integer
Page number this response covers (1-based).
pagination.nextPageUrl
string | null
URL for the next page of results, or null on the last page.
pagination.hasNext
boolean
True when a next page is available.

Sample response

{
  "site": "stackoverflow.com",
  "query": "web scraping infinite scroll pagination",
  "tag": null,
  "sort": "Relevance",
  "currentPage": 1,
  "totalQuestions": null,
  "results": [
    {
      "position": 1,
      "id": "78912345",
      "title": "How to handle pagination when scraping an infinite scroll page?",
      "questionUrl": "https://stackoverflow.com/questions/78912345/how-to-handle-pagination-when-scraping-an-infinite-scroll-page",
      "score": 4,
      "answerCount": 2,
      "viewCount": 137,
      "hasAcceptedAnswer": true,
      "excerpt": "I'm trying to scrape a product listing that loads more items as you scroll. The network tab shows a POST request returning JSON, but the cursor token keeps changing...",
      "tags": ["python", "web-scraping", "pagination", "requests"],
      "author": "dev_ana",
      "authorUrl": "https://stackoverflow.com/users/1234567/dev-ana",
      "askedAt": "2026-07-15T09:42:11Z"
    },
    {
      "position": 2,
      "id": "78911002",
      "title": "Rotating proxies with residential IPs to avoid rate limiting",
      "questionUrl": "https://stackoverflow.com/questions/78911002/rotating-proxies-with-residential-ips-to-avoid-rate-limiting",
      "score": 1,
      "answerCount": 0,
      "viewCount": 42,
      "hasAcceptedAnswer": false,
      "excerpt": "My crawler gets blocked after roughly 500 requests from a datacenter IP. I want to rotate through a residential proxy pool but I'm unsure how to detect a soft ban...",
      "tags": ["web-scraping", "proxy", "http"],
      "author": "crawler_joe",
      "authorUrl": "https://stackoverflow.com/users/9988776/crawler-joe",
      "askedAt": "2026-07-15T08:05:47Z"
    }
  ],
  "resultCount": 2,
  "pagination": {
    "currentPage": 1,
    "nextPageUrl": "https://stackoverflow.com/search?q=web+scraping+infinite+scroll+pagination&tab=Relevance&page=2",
    "hasNext": true
  }
}