How to Scrape Bing Search Resultsa Python walkthrough

Bing is the second most used search engine in the world and the default on a large share of Windows desktops, so its public results page is a useful second opinion alongside Google. For keyword research, SEO rank tracking, market analysis, or feeding a model with real-world search data, the Bing SERP carries exactly the structured signal you want: titles, links, snippets, and the order they appear in.

This guide shows you how to scrape Bing search results with Python the reliable way. You build a small, runnable scraper that fetches a rendered results page through the Crawling API, parses each organic result with BeautifulSoup, handles pagination, and exports the data to JSON and CSV. The whole walkthrough stays scoped to public search-results data that anyone can see without an account, and the legality section near the end is worth reading before you point this at any real volume.

What you will build

A Python script that takes a public Bing search URL, retrieves the HTML through the Crawling API, and extracts a structured record for every organic result on the page. We will use a sample query as the running example and pull these fields from each result:

• Position the rank of the result on the page, counted from the top.
• Title the clickable headline text of the result, as shown in the listing.
• Link the destination URL the result points to.
• Description the displayed snippet or summary under the title.

On top of the per-result fields, the script walks across multiple result pages and writes everything out to both a JSON file and a CSV file, so the data drops straight into a spreadsheet or a database.

Why a plain request fails on Bing

If you fire a bare HTTP request at a Bing results URL from a script, you rarely get the clean page you see in your own browser. Two things work against you. First, Bing leans on JavaScript to assemble parts of the results page, so a raw fetch can come back with a skeleton that is missing the listings you actually want. Second, Bing watches for automated traffic: requests that do not look like a real browser get challenged, fed a verification page, or throttled before they reach the results.

So a working Bing scraper needs two things in one request: an IP the platform reads as a real visitor, and, when the page leans on scripts, a browser that renders it. You can assemble that yourself with a headless browser plus a pool of rotating residential proxies, but keeping those healthy is most of the work. The Crawling API folds both into a single call: you send it the URL, it fetches from a trusted IP and renders when needed, and it returns finished HTML for you to parse.

Prerequisites

You need a few things in place before writing any code. None of them take long.

Basic Python. You should be comfortable writing and running a Python script and installing packages with pip. If BeautifulSoup is new to you, our guide to using BeautifulSoup in Python covers the parsing basics this tutorial assumes.

Python 3.8 or later. Confirm your version with python --version. If you do not have it, install it from python.org or through a distribution like Anaconda.

A Crawlbase account and token. Sign up, open your dashboard, and copy your request token from the account docs page. Your first 1,000 requests are free, and adding billing details before you spend them unlocks an extra 9,000 free requests. Treat the token like a password: it authenticates your requests, so keep it out of version control.

Set up the project

Create a virtual environment so project dependencies stay isolated, then install the two libraries the scraper needs.

python --version

python -m venv bing_env
source bing_env/bin/activate

pip install requests beautifulsoup4

On Windows, activate the environment with bing_env\Scripts\activate instead of the source line. Two dependencies do the work: requests sends the HTTP call to the Crawling API, and beautifulsoup4 parses the returned HTML so you can pull out individual fields by CSS selector.

Step 1: Fetch the page through the Crawling API

Start by getting the HTML. Write a small crawl() function that sends your target URL to the Crawling API with your token, asks for JavaScript rendering so the full listing loads, checks that the underlying page came back with a 200 status, and returns the HTML body. Checking the status before you parse keeps failures loud instead of silent.

import json
import requests

API_TOKEN = "YOUR_CRAWLBASE_TOKEN"  # replace with your token
API_ENDPOINT = "https://api.crawlbase.com/"

def crawl(url):
    params = {"token": API_TOKEN, "url": url, "javascript": "true"}
    response = requests.get(API_ENDPOINT, params=params)
    response.raise_for_status()

    data = json.loads(response.text)
    if data["original_status"] != 200:
        raise Exception(f"Unable to crawl '{url}'")

    return data["body"]

if __name__ == "__main__":
    url = "https://www.bing.com/search?q=samsung+s23+ultra"
    html = crawl(url)
    print(html[:500])

The API returns a JSON envelope, so you load the response with json.loads and read two fields: original_status is the status Bing itself returned, and body is the page HTML. The javascript=true parameter tells the API to render the page in a real browser before handing it back, which is what guarantees the full result list is present. Guarding on original_status means a block or a challenge surfaces as an exception instead of feeding garbage into the parser. The sample query "samsung s23 ultra" rides in the q parameter, which is how Bing carries the search term. Run the script with python crawling.py and you should see real results markup in the first 500 characters, which confirms the fetch works before you write a single selector.

Step 2: Parse the results with BeautifulSoup

With HTML in hand, load it into BeautifulSoup and pull each result by its selector. Bing wraps every organic result in an li.b_algo list item, with the title and link in an h2 a anchor and the snippet in a p.b_algoSlug paragraph. Inspect the live page in your browser's dev tools (right-click, then Inspect) to confirm the current class names; the selectors below match Bing's layout at the time of writing.

from bs4 import BeautifulSoup

def scrape_html(html):
    soup = BeautifulSoup(html, "html.parser")

    results = []
    for position, block in enumerate(soup.select("li.b_algo"), start=1):
        link = block.select_one("h2 a")
        snippet = block.select_one("p.b_algoSlug")
        if not link:
            continue
        results.append({
            "position": position,
            "title": link.get_text(strip=True),
            "url": link.get("href"),
            "description": snippet.get_text(strip=True) if snippet else None,
        })

    return results

The selector li.b_algo is the container Bing uses for each organic result, so looping over those list items gives you exactly the listings and skips the page chrome. Reading the title text and the href from the same h2 a anchor keeps the headline and the destination link aligned, and p.b_algoSlug holds the description shown under each title. enumerate(..., start=1) gives you the position for free as you loop, so rank comes from page order instead of a fragile attribute. The if not link: continue guard skips any block that has no headline anchor, which keeps ads, video carousels, and stray markup out of your output. The snippet falls back to None when a result has no description paragraph.

Step 3: Put it together

Now wire the fetch and the parse into one runnable script. Crawl the rendered results page, hand the HTML to the parser, print the results, and write the structured output to JSON.

import json
import requests
from bs4 import BeautifulSoup

API_TOKEN = "YOUR_CRAWLBASE_TOKEN"
API_ENDPOINT = "https://api.crawlbase.com/"

def crawl(url):
    params = {"token": API_TOKEN, "url": url, "javascript": "true"}
    response = requests.get(API_ENDPOINT, params=params)
    response.raise_for_status()
    data = json.loads(response.text)
    if data["original_status"] != 200:
        raise Exception(f"Unable to crawl '{url}'")
    return data["body"]

def scrape_html(html):
    soup = BeautifulSoup(html, "html.parser")
    results = []
    for position, block in enumerate(soup.select("li.b_algo"), start=1):
        link = block.select_one("h2 a")
        snippet = block.select_one("p.b_algoSlug")
        if not link:
            continue
        results.append({
            "position": position,
            "title": link.get_text(strip=True),
            "url": link.get("href"),
            "description": snippet.get_text(strip=True) if snippet else None,
        })
    return results

def main():
    url = "https://www.bing.com/search?q=samsung+s23+ultra"
    html = crawl(url)
    results = scrape_html(html)
    print(json.dumps(results, indent=2, ensure_ascii=False))
    with open("bing_results.json", "w", encoding="utf-8") as f:
        json.dump(results, f, ensure_ascii=False, indent=2)
    print(f"Saved {len(results)} results")

if __name__ == "__main__":
    main()

Run the full script with python main.py. It fetches the results page for "samsung s23 ultra", extracts a record for each organic listing, prints them, and writes everything to bing_results.json. The same two functions are all you need: swap the query in the URL and the parser handles whatever comes back.

What the output looks like

You get a clean ordered list of result objects, each with a position, title, link, and description, ready to write to JSON, CSV, or a database.

[
  {
    "position": 1,
    "title": "Samsung Galaxy S23 Ultra | Samsung US",
    "url": "https://www.samsung.com/us/smartphones/galaxy-s23-ultra/",
    "description": "Meet the latest Galaxy S23 Ultra phone, equipped with a built-in S Pen, Nightography camera, and a powerful chip for epic gaming."
  },
  {
    "position": 2,
    "title": "Samsung Galaxy S23 Ultra - Full phone specifications",
    "url": "https://www.gsmarena.com/samsung_galaxy_s23_ultra-12024.php",
    "description": "Samsung Galaxy S23 Ultra Android smartphone. Announced Feb 2023. Features 6.8 inch display, Snapdragon 8 Gen 2 chipset, 5000 mAh battery."
  }
]

Scaling across pages and queries

One query on one page is a demo; a real job runs over several searches and deeper into the results. Bing paginates with the first query parameter, which is a 1-indexed offset in steps of 10: first=11 is the second page, first=21 the third, and so on. The shape stays the same: build each URL, fetch it through the Crawling API, and parse it with the same function. The one habit that keeps a long run healthy is pacing, so pause between requests rather than firing them in a tight loop.

import time
from urllib.parse import quote_plus

query = "samsung s23 ultra"
encoded = quote_plus(query)

all_results = []
for page in range(3):
    first = page * 10 + 1
    url = f"https://www.bing.com/search?q={encoded}&first={first}"
    html = crawl(url)
    all_results.extend(scrape_html(html))
    time.sleep(3)

print(f"Collected {len(all_results)} results across 3 pages")

Crawlbase serves up to 20 requests per second by default, which is plenty of headroom for a scraper that paces itself; if you genuinely need more, support can raise it. Any 5XX response from the API is free of charge, so retrying a blocked or unavailable URL costs you nothing. If you would rather route your own traffic through a rotating pool instead of using the managed API, the Smart AI Proxy (also called the AI Proxy) gives you the same residential IP rotation as a drop-in proxy endpoint.

Export the results to CSV

JSON is convenient for code, but a CSV opens straight in a spreadsheet, so it is the format most teams hand around. Because every result is already a flat dictionary with the same keys, writing a CSV takes a few lines with the standard library.

import csv

def save_csv(results, path="bing_results.csv"):
    fields = ["position", "title", "url", "description"]
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fields)
        writer.writeheader()
        writer.writerows(results)
    print(f"Wrote {len(results)} rows to {path}")

Call save_csv(all_results) after a paginated run and you get a bing_results.csv with one row per organic result and a header line on top. Using DictWriter with an explicit fieldnames list keeps the column order stable, and newline="" stops the file from picking up blank lines on Windows.

Staying unblocked

Even with a trusted IP and rendering handled, Bing watches for scraper-shaped traffic. A few habits keep a run healthy.

• Pace your requests. Hammering results pages in a tight loop is the fastest way to get challenged. Spread requests out and vary your queries instead of paging one term at full speed.
• Lean on rotation. A pool of residential IPs spreads requests across many real-user addresses so no single one trips a limit. The Crawling API handles this for you; if you roll your own stack, this is the part to get right.
• Read the status codes. A run that starts returning challenges or verification pages is telling you the current rate or IP tier is no longer enough. Treat that as signal to back off, not noise to ignore.
• Re-inspect when fields go empty. Bing changes its markup periodically. If results stop parsing, open a live page in dev tools and update the selectors.

For the broader playbook, see how to scrape websites without getting blocked. If a Bing page you need leans on scripts to render, our guide on crawling JavaScript websites explains why rendering matters and how to turn it on. For a wider view of how the major engines structure their pages, see what Google, Yahoo, and Bing return.

Is it legal to scrape Bing?

Whether scraping Bing is allowed depends on Microsoft's terms of service, your jurisdiction, and what you do with the data. Bing's terms place limits on automated access, so scraping can run against those terms regardless of how careful your tooling is. None of the code here changes that; it just makes the technical part work. Read Bing's terms and its robots.txt, and treat both as the boundary for what you collect.

A few lines worth holding to. Collect only public search-results data: the titles, links, descriptions, and positions that anyone can see on a results page without an account. Keep your request volume low enough that you are not straining Bing's servers, and pace your crawl rather than running it flat out. If you need search data at scale and want a sanctioned path, Microsoft offers the Bing Search APIs through Azure, which is the official, supported way to query Bing results programmatically and is the right choice when your project needs volume or guarantees.

This guide is deliberately scoped to public search-results pages because that is the line that keeps the work defensible. It does not cover anything behind a login, account or personal data, or copyrighted media pulled from the linked destinations. Public SERP data only. If your project needs more than that, an official data agreement or the Azure Bing Search API is the correct path, not a cleverer scraper.

Frequently Asked Questions (FAQs)

Why does a plain request fail or return the wrong page on Bing?

Bing leans on JavaScript to assemble parts of the results page, so a raw fetch can come back with a skeleton that is missing the listings. It also flags traffic that does not look like a real browser and can answer with a challenge or a verification page. Fetching through the Crawling API with javascript=true, which renders the page from a rotating residential IP, makes the request look like an ordinary visitor so you get the real results page.

Can I scrape Bing search results with Python?

Yes. With requests and BeautifulSoup you can fetch a results page and pull out titles, links, descriptions, and positions. The Crawling API acts as the bridge that gets your request to Bing from a trusted IP and renders it, so requests are processed smoothly instead of being blocked. For a broader Python primer, see our guide on scraping websites with Python.

What fields can I extract from a Bing results page?

This tutorial pulls four fields from each organic result: the position on the page, the title, the destination link, and the displayed description. The title and link come from the h2 a anchor inside each li.b_algo block, and the description comes from p.b_algoSlug. Stay within public search-results data and avoid anything behind a login.

Do I need JavaScript rendering to scrape Bing?

Often yes, because Bing fills in parts of the results page with scripts. The examples here pass javascript=true to the Crawling API so the page is rendered in a real browser before it comes back, which guarantees the full listing is present. Our guide to scraping JavaScript pages with Python covers when rendering is necessary.

How do I paginate through more Bing results?

Use the first query parameter, which is a 1-indexed offset in steps of 10: first=11 is the second page, first=21 the third, and so on. Build each page URL with the offset, fetch it through the Crawling API, parse it with the same function, and pause a few seconds between requests so you are pacing the crawl rather than hammering it.

Is scraping Bing different from scraping Google?

The approach is the same, only the selectors and pagination parameters differ. Bing uses li.b_algo blocks and a first offset, while Google uses its own result containers and a start offset. If you also work with Google, see our guides on scraping Google search pages and on scraping Yandex search results for the equivalent selectors and steps.

MA

Muhammad Atif

Senior Full Stack Developer · Crawlbase

Senior full stack developer at Crawlbase, building the platform and writing about scraping architecture, proxies, and data pipelines.

IK

Ian Kalvin

Technical Support Engineer · Crawlbase

Technical support engineer at Crawlbase, writing from the front line of what actually breaks in production scraping and proxy setups.