Stack Exchange Thread · Crawlbase Documentation

API usage

Add &scraper=stackexchange-thread to a Crawling API request. URL-encode the target URL in the url parameter.

curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \
  --data-urlencode 'url=https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file' \
  --data-urlencode 'scraper=stackexchange-thread' -G
from crawlbase import CrawlingAPI

api = CrawlingAPI({'token': 'YOUR_TOKEN'})
res = api.get(
    'https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file',
    {'scraper': 'stackexchange-thread'}
)

import json
data = json.loads(res['body'])
const { CrawlingAPI } = require('crawlbase');
const api = new CrawlingAPI({ token: 'YOUR_TOKEN' });

const res = await api.get(
  'https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file',
  { scraper: 'stackexchange-thread' }
);
const data = JSON.parse(res.body);
require 'crawlbase'
api = Crawlbase::API.new(token: 'YOUR_TOKEN')

res = api.get('https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file', scraper: 'stackexchange-thread')
data = JSON.parse(res.body)

Example input URL

Any Stack Exchange question URL works in the url parameter, from any site in the network. For example:

https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file
https://superuser.com/questions/441895/automate-opening-html-and-printing-to-pdf
https://askubuntu.com/questions/1206658/why-is-apt-held-back
https://serverfault.com/questions/439471/which-openvpn-cipher-should-i-use
https://mathoverflow.net/questions/44265/is-the-riemann-hypothesis

Response shape

JSON response body. Field types may be null when the source page omits the value.

question

object

The question itself.

question.id

string

Stack Exchange question id.

question.title

string

Question title.

question.url

string

Canonical URL of the question.

question.body

string

Question body text.

question.score

integer

Question score (upvotes minus downvotes).

question.viewCount

integer

Number of views on the question.

question.tags

array

Tags applied to the question.

question.askedAt

string

Question creation time (ISO 8601).

question.author

object

Author of the question.

question.author.name

string

Display name of the question author.

question.author.url

string

Profile URL of the question author.

question.author.reputation

integer

Reputation score of the question author.

question.comments

array

Comments on the question.

question.comments[].id

string

Comment id.

question.comments[].score

integer

Comment score.

question.comments[].body

string

Comment text.

question.comments[].author

string

Display name of the comment author.

question.comments[].authorUrl

string

Profile URL of the comment author.

question.comments[].createdAt

string

Comment creation time (ISO 8601).

answerCount

integer

Number of answers returned in answers.

answers

array

Answers to the question, in listing order.

answers[].id

string

Answer id.

answers[].score

integer

Answer score (upvotes minus downvotes).

answers[].isAccepted

boolean

True when this is the accepted answer.

answers[].body

string

Answer body text.

answers[].author

object

Author of the answer.

answers[].author.name

string

Display name of the answer author.

answers[].author.url

string

Profile URL of the answer author.

answers[].author.reputation

integer

Reputation score of the answer author.

answers[].createdAt

string

Answer creation time (ISO 8601).

answers[].comments

array

Comments on the answer.

answers[].comments[].id

string

Comment id.

answers[].comments[].score

integer

Comment score.

answers[].comments[].body

string

Comment text.

answers[].comments[].author

string

Display name of the comment author.

answers[].comments[].authorUrl

string

Profile URL of the comment author.

answers[].comments[].createdAt

string

Comment creation time (ISO 8601).

Sample response

{
  "question": {
    "id": "2861071",
    "title": "How to modify a text file?",
    "url": "https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file",
    "body": "I'm using Python and I need to insert a line at the start of a text file without loading the whole file into memory. Is there a way to do this in place, or do I have to rewrite the file?",
    "score": 312,
    "viewCount": 486201,
    "tags": ["python", "file", "text-files"],
    "askedAt": "2010-05-18T20:11:33Z",
    "author": {
      "name": "Nathan Fellman",
      "url": "https://stackoverflow.com/users/8460/nathan-fellman",
      "reputation": 127843
    },
    "comments": [
      {
        "id": "2954120",
        "score": 3,
        "body": "Do you need to preserve the original file, or is rewriting acceptable?",
        "author": "Greg Hewgill",
        "authorUrl": "https://stackoverflow.com/users/893/greg-hewgill",
        "createdAt": "2010-05-18T20:19:04Z"
      }
    ]
  },
  "answerCount": 2,
  "answers": [
    {
      "id": "2861108",
      "score": 401,
      "isAccepted": true,
      "body": "You cannot insert into the middle of a file without rewriting it. Read the file into a list of lines, insert your new line, then write it all back:\n\n    with open('file.txt') as f:\n        lines = f.readlines()\n    lines.insert(0, 'new first line\\n')\n    with open('file.txt', 'w') as f:\n        f.writelines(lines)\n",
      "author": {
        "name": "Roberto Bonvallet",
        "url": "https://stackoverflow.com/users/193568/roberto-bonvallet",
        "reputation": 30215
      },
      "createdAt": "2010-05-18T20:15:52Z",
      "comments": [
        {
          "id": "2954260",
          "score": 12,
          "body": "For very large files, stream through a temporary file instead of holding everything in memory.",
          "author": "John Machin",
          "authorUrl": "https://stackoverflow.com/users/253537/john-machin",
          "createdAt": "2010-05-18T21:02:11Z"
        }
      ]
    },
    {
      "id": "2861180",
      "score": 47,
      "isAccepted": false,
      "body": "If the file is large, use fileinput with inplace=True to edit it line by line without loading it all at once.",
      "author": {
        "name": "codeape",
        "url": "https://stackoverflow.com/users/18770/codeape",
        "reputation": 98412
      },
      "createdAt": "2010-05-18T20:24:39Z",
      "comments": []
    }
  ]
}