How to Download Images Using Pythonsix methods that scale

Images are some of the heaviest and most useful files on the web: product photos for a catalog, charts for a research dataset, assets for a machine learning pipeline. When you need more than a handful, clicking "Save image as" stops being an option, and a short Python script does the job in seconds instead of an afternoon.

This guide walks through six practical ways to download images using Python, from a single file with requests to pulling every image off a page with BeautifulSoup, streaming large files in chunks, organising what you save, and reaching protected sources through the Crawlbase Crawling API. Every snippet is real and runs as written, so you can copy each one and adapt it to your own targets.

What you will build

By the end you will have a small toolkit of functions that cover the common cases, plus a short script that ties them together.

• Single download. Fetch one image by URL and write it to disk with requests.
• Standard-library download. Do the same with no third-party packages using urllib.request.
• Page scrape. Find every <img> on a page with BeautifulSoup and download each source.
• Chunked streaming. Save large files in pieces so memory use stays flat.
• Naming and folders. Derive safe filenames and sort downloads into directories.
• Protected sources. Reach images behind rendering or bot protection through the Crawling API.

Why a plain download fails on some sites

Downloading an image is, at its simplest, one HTTP GET. That works perfectly for static files served from a predictable URL. The trouble starts on real sites. Some pages build their image grid with JavaScript after the initial HTML loads, so a bare fetch returns markup with no image tags in it. Others sit behind anti-bot defenses that challenge or block requests coming from datacenter IPs or anything that does not look like a real browser, and you get a 403 or an HTML error page where you expected a JPEG.

For straightforward files the first five methods below are all you need. For the awkward cases, the last method routes the request through a rendering layer and a pool of trusted IPs so the file comes back intact. We cover the simple path first because most downloads never need more than that.

Prerequisites

You do not need much to follow along.

Python 3.8 or later. Check your version with python --version. If you do not have it, install it from python.org.

Basic Python. You should be comfortable running a script and installing packages with pip. Functions, loops, and the with statement are enough.

A Crawlbase account (for the last method only). The first five methods use only requests, urllib, and BeautifulSoup. For the protected-source method you will need a free Crawlbase account and its API token.

Set up the project

Create a virtual environment so the project's dependencies stay isolated, then install the two third-party libraries used in this guide.

python --version

python -m venv image_env
source image_env/bin/activate

pip install requests beautifulsoup4

On Windows, activate the environment with image_env\Scripts\activate instead of the source line. requests is the HTTP client that fetches each file, and beautifulsoup4 parses page HTML so you can find image tags. urllib, os, and hashlib are part of the standard library and need no install.

Tip 1: Download a single image with requests

The most common case is one image at a given URL. Send a GET, confirm the response came back as an image, and write the bytes to a file in binary mode. Checking the status code and content type before you write keeps you from saving an HTML error page under a .jpg name.

import requests

url = "https://www.python.org/static/img/python-logo.png"
headers = {"User-Agent": "Mozilla/5.0 (image downloader)"}

response = requests.get(url, headers=headers, timeout=10)
if response.status_code == 200 and "image" in response.headers.get("Content-Type", ""):
    with open("python-logo.png", "wb") as f:
        f.write(response.content)
    print("Saved python-logo.png")
else:
    print(f"Skipped: {response.status_code} {response.headers.get('Content-Type')}")

Three details matter here. The file is opened with "wb" (write binary) because image data is bytes, not text, and writing it as text would corrupt the file. The Content-Type check confirms the server actually returned an image rather than an error page that happens to carry a 200 status. The timeout stops the script from hanging forever on a stalled server. Run this and you should see Saved python-logo.png with a real PNG on disk next to your script. That is a working download.

Tip 2: Download with urllib from the standard library

If you would rather not add a dependency, the standard library can do the same job. urllib.request ships with Python, so this approach needs nothing installed at all. Setting a User-Agent through a Request object helps with servers that reject the default urllib agent.

import urllib.request

url = "https://www.python.org/static/img/python-logo.png"
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 (image downloader)"})

with urllib.request.urlopen(req, timeout=10) as resp, open("logo_urllib.png", "wb") as f:
    f.write(resp.read())

print("Saved logo_urllib.png")

The with statement opens both the connection and the output file together and closes them cleanly when the block ends, even if an error is raised mid-write. There is also a one-line shortcut, urllib.request.urlretrieve(url, "logo.png"), which is handy for quick scripts but gives you no control over headers or error handling, so the explicit form above is the safer default. Both requests and urllib get the same bytes onto disk; requests simply has a friendlier API, which is why most of the rest of this guide uses it.

Tip 3: Download every image on a page with BeautifulSoup

Downloading one file is easy. The real work is pulling every image off a page automatically. The pattern is two steps: fetch the page HTML, then parse it with BeautifulSoup to collect the src of every <img> tag, and finally loop over those URLs reusing the single-download logic from Tip 1.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

page_url = "https://en.wikipedia.org/wiki/Python_(programming_language)"
headers = {"User-Agent": "Mozilla/5.0 (image downloader)"}

page = requests.get(page_url, headers=headers, timeout=10)
soup = BeautifulSoup(page.text, "html.parser")

image_urls = []
for img in soup.select("img"):
    src = img.get("src")
    if src:
        image_urls.append(urljoin(page_url, src))

print(f"Found {len(image_urls)} images")

Two parts of this are worth slowing down on. The selector img grabs every image tag on the page, and reading img.get("src") instead of img["src"] returns None for any tag missing the attribute rather than raising an error. The other key piece is urljoin: image sources are often relative paths like /images/photo.jpg, and joining each one against the page URL turns it into a complete, downloadable address. For a deeper tour of selecting elements this way, see how to use BeautifulSoup in Python.

With the list of absolute URLs in hand, the download loop reuses the binary-write pattern from Tip 1 and gives each file its own name.

import os

os.makedirs("downloads", exist_ok=True)

for i, img_url in enumerate(image_urls):
    try:
        r = requests.get(img_url, headers=headers, timeout=10)
        if r.status_code == 200 and "image" in r.headers.get("Content-Type", ""):
            path = os.path.join("downloads", f"image_{i}.jpg")
            with open(path, "wb") as f:
                f.write(r.content)
    except requests.RequestException as e:
        print(f"Failed {img_url}: {e}")

The try/except around each request is not optional once you are downloading many files. Out of dozens of images, one will eventually time out, return a redirect loop, or vanish, and catching requests.RequestException lets the loop skip the bad one and keep going instead of crashing on file thirty. The os.makedirs(..., exist_ok=True) call creates the output folder once and does nothing if it already exists.

Tip 4: Stream large files in chunks

Reading response.content loads the entire file into memory before writing it. That is fine for a logo, but it is wasteful for a high-resolution photo or a multi-megabyte asset, and it can exhaust memory when you download many large files in a row. Streaming the response and writing it in fixed-size chunks keeps memory use flat regardless of file size.

def download_stream(url, path, chunk_size=8192):
    with requests.get(url, headers=headers, stream=True, timeout=30) as r:
        r.raise_for_status()
        with open(path, "wb") as f:
            for chunk in r.iter_content(chunk_size=chunk_size):
                f.write(chunk)
    return path

download_stream(
    "https://upload.wikimedia.org/wikipedia/commons/c/c3/Python-logo-notext.svg",
    "downloads/large_logo.svg",
)

Passing stream=True tells requests not to download the body up front; instead iter_content pulls it in pieces of chunk_size bytes and writes each piece straight to disk. Peak memory stays around the size of one chunk, eight kilobytes here, no matter how large the file is. raise_for_status() turns a 4xx or 5xx response into an exception so you do not silently save an error body, and the outer with on the request ensures the connection is released once the file is written.

Tip 5: Name and organise your files

Saving everything as image_0.jpg, image_1.jpg works, but it discards the original filenames and loses the file extension, which matters if you are mixing PNGs, JPEGs, and SVGs. A small helper derives a clean name from the URL and falls back to a content hash when the URL has no usable name, which guarantees uniqueness and avoids overwriting two files that happen to share a name.

import os
import hashlib
from urllib.parse import urlparse

def filename_for(url, content, folder="downloads"):
    name = os.path.basename(urlparse(url).path)
    if not name or "." not in name:
        digest = hashlib.md5(content).hexdigest()[:12]
        name = f"{digest}.jpg"
    return os.path.join(folder, name)

# Example: turn a messy URL into a tidy path
r = requests.get(image_urls[0], headers=headers, timeout=10)
print(filename_for(image_urls[0], r.content))

urlparse(...).path strips query strings and fragments off the URL, and os.path.basename takes just the final segment, so .../photo.jpg?size=large becomes photo.jpg. When the URL has no real filename, the MD5 hash of the file's bytes gives a short, stable, collision-resistant name. The same hash is also the simplest way to spot duplicates: two identical images produce the same digest, so you can skip a file you have already saved before writing it.

Tip 6: Download from protected sources with the Crawling API

The five methods above cover any image you can reach with a plain request. Some sites are not that cooperative. The page may render its image grid with JavaScript so a fetch returns no <img> tags, or the image host may block datacenter IPs and serve a 403 instead of the file. You can solve both yourself by running a headless browser and maintaining a pool of rotating residential proxies, but building and keeping that healthy is most of the engineering effort and has nothing to do with the images you want.

The Crawling API folds rendering and IP rotation into a single request. You install the official client, then route the page fetch through it; the returned HTML is fully rendered, so the BeautifulSoup parsing from Tip 3 works unchanged.

pip install crawlbase

from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup
from urllib.parse import urljoin

api = CrawlingAPI({"token": "YOUR_CRAWLBASE_JS_TOKEN"})
page_url = "https://example.com/gallery"

result = api.get(page_url, {"ajax_wait": "true", "page_wait": 5000})
html = result["body"].decode("utf-8") if result["status_code"] == 200 else None

# Same parsing as Tip 3, now against rendered HTML
soup = BeautifulSoup(html, "html.parser")
image_urls = [urljoin(page_url, img["src"]) for img in soup.select("img[src]")]
print(f"Found {len(image_urls)} images on the rendered page")

The two wait options matter on a client-rendered gallery. ajax_wait tells the API to wait for asynchronous content to finish loading, and page_wait holds for a fixed number of milliseconds after load so late images appear before capture. Use the JavaScript token for sites that build their grid in the browser; for static image hosts the normal token is faster. Once image_urls is populated, you feed it straight into the chunked download_stream helper from Tip 4. If you only need the IP rotation and not the rendering, for example when the page HTML is fine but the image host blocks you, the Smart AI Proxy routes ordinary requests traffic through the same trusted pool with a one-line change to your proxy settings.

What the output looks like

After a page scrape, a small manifest of what you saved is useful for auditing and for skipping duplicates on the next run. Writing one record per file gives you a structured log alongside the images themselves.

[
  {
    "source_url": "https://example.com/img/photo-01.jpg",
    "saved_as": "downloads/photo-01.jpg",
    "content_type": "image/jpeg",
    "bytes": 184213
  },
  {
    "source_url": "https://example.com/img/diagram.svg",
    "saved_as": "downloads/diagram.svg",
    "content_type": "image/svg+xml",
    "bytes": 9042
  }
]

Each record ties the original URL to the local file, notes the content type so you know the real format, and records the byte size. Keeping the source_url means a later run can check what you already have before downloading it again.

Scaling to many images

For a few dozen files the sequential loop in Tip 3 is fine. When you are pulling thousands, downloading one at a time is the bottleneck, because most of the time is spent waiting on the network rather than on your CPU. A thread pool lets several downloads run concurrently while still being far simpler than full async code.

from concurrent.futures import ThreadPoolExecutor

def save_one(url):
    try:
        r = requests.get(url, headers=headers, timeout=15)
        if r.status_code == 200 and "image" in r.headers.get("Content-Type", ""):
            path = filename_for(url, r.content)
            with open(path, "wb") as f:
                f.write(r.content)
            return path
    except requests.RequestException:
        return None

with ThreadPoolExecutor(max_workers=8) as pool:
    saved = list(pool.map(save_one, image_urls))

print(f"Saved {len([p for p in saved if p])} of {len(image_urls)} images")

Eight workers is a sensible starting point; push it too high and you risk hammering the host, which is both impolite and a fast route to getting rate-limited. Each save_one call is self-contained and swallows its own errors so one failure never sinks the batch. If you are downloading at this volume from a protected source, route save_one through the Crawling API or the Smart AI Proxy so the rotating IPs absorb the load instead of your single address. For more on keeping large runs alive, see how to scrape websites without getting blocked.

Downloading images responsibly

Images are not free-floating data; they are creative work, and almost every image you find online is copyrighted by whoever made it. Being able to download a file is not the same as being allowed to use it. Before you point a script at a site, read its terms of service and check its robots.txt, keep your request volume low enough that you are not straining the server, and download only what you actually have the right to use. Photos behind a login, personal images, and stock you have not licensed are off limits.

The same care applies to what you do afterward. Do not redistribute someone else's images as your own, and do not feed copyrighted media into a model or dataset without permission from the rights holder. When a site offers an official API or a licensing path for its media, use it: it is the cleanest way to get images you can rely on legally, and it usually comes with terms that spell out exactly what you may do with them.

Frequently Asked Questions (FAQs)

What is the simplest way to download an image in Python?

Send a GET with requests, then write response.content to a file opened in binary mode ("wb"). That is three lines for a single file. Check the status code and the Content-Type header first so you only save real image data and not an HTML error page that happened to return a 200 status.

Should I use requests or urllib?

Both download the same bytes. requests has a cleaner API, easier header handling, and built-in streaming, which is why most code uses it. Reach for urllib.request when you want zero third-party dependencies, since it ships with Python. The download logic is otherwise identical.

How do I download all images from a web page?

Fetch the page HTML, parse it with BeautifulSoup, and collect the src of every <img> tag. Resolve relative paths to absolute URLs with urljoin, then loop over the list and download each file with the single-image pattern. Wrap each download in try/except so one bad URL does not stop the whole batch, as shown in Tip 3.

Why does my downloaded file open as a broken image?

Usually one of two reasons. Either the file was opened in text mode instead of binary, which corrupts the bytes, so make sure you use "wb". Or the server returned an HTML error page rather than the image, which is why checking the status code and Content-Type before writing matters. On JavaScript-heavy pages the image may not exist in the initial HTML at all, in which case you need a rendering step.

How do I download images without getting blocked?

Send a realistic User-Agent, pace your requests with a short delay, and avoid hammering one host with many concurrent threads. At higher volume you also need IPs that read as real visitors, which a single machine cannot provide. Routing through rotating residential IPs, via the Crawling API or the Smart AI Proxy, is what keeps large image runs from tripping rate limits.

How can I avoid downloading the same image twice?

Compute a hash of each file's bytes with hashlib and keep the digests you have already seen in a set. Before writing a new file, check whether its hash is already in the set; if it is, skip it. The same digest also makes a reliable, collision-resistant filename when the URL has no usable name of its own.

BA

Bilal Ahmed

Software Engineer · Crawlbase

Software engineer who wrote some of the most-read pieces on the Crawlbase blog, covering web scraping, proxies, and data tooling.