Web Scraping with Python: Scraping Images from a Website Using BeautifulSoup

Images are an important part of many websites, and sometimes, you may want to scrape and download images for use in your own projects or for analysis. Python makes this task easy with the help of BeautifulSoup and requests. In this article, we will demonstrate how to scrape images from a website and save them to your local machine using these two powerful Python libraries.

Step 1: Install Required Libraries

Before we begin scraping, we need to install the necessary libraries: requests for handling HTTP requests and BeautifulSoup for parsing the HTML content.

To install the required libraries, run:

pip install requests beautifulsoup4

Step 2: Find Image URLs

First, let's identify the URLs of the images we want to scrape. Using BeautifulSoup, we can parse the HTML of a webpage and find all image elements.

Here's how you can extract image URLs from a webpage:

import requests
from bs4 import BeautifulSoup

# Define the URL of the website
url = "https://example.com"

# Send a GET request to the website
response = requests.get(url)

# Check if the request was successful
if response.status_code == 200:
    print("Page loaded successfully!")
    soup = BeautifulSoup(response.text, "html.parser")
    
    # Find all image tags on the page
    images = soup.find_all("img")
    
    # Extract image URLs (the 'src' attribute of each image)
    image_urls = [img["src"] for img in images if "src" in img.attrs]
    
    print(f"Found {len(image_urls)} images.")
else:
    print("Failed to retrieve the webpage.")

This script will find all img tags in the HTML and extract the src attribute, which contains the URL of the image.

Step 3: Handle Relative and Absolute URLs

In some cases, the src attribute may contain relative URLs, which need to be converted to absolute URLs. To handle this, you can use Python’s urljoin function to ensure that all image URLs are absolute.

Here’s how to do it:

from urllib.parse import urljoin

# Convert relative URLs to absolute URLs
absolute_image_urls = [urljoin(url, img_url) for img_url in image_urls]

# Print out the absolute URLs
for img_url in absolute_image_urls:
    print(img_url)

Step 4: Download Images

Once we have the image URLs, we can download the images to our local machine. We'll use the requests library to fetch the image data and save it to a file.

Here’s an example of how to download the images:


import os

# Create a directory to store the images
os.makedirs("scraped_images", exist_ok=True)

# Download each image
for img_url in absolute_image_urls:
    try:
        # Send a GET request to fetch the image
        img_response = requests.get(img_url)
        
        # Check if the image was fetched successfully
        if img_response.status_code == 200:
            # Extract the image name from the URL (using the last part of the URL)
            img_name = os.path.basename(img_url)
            
            # Define the path to save the image
            img_path = os.path.join("scraped_images", img_name)
            
            # Save the image to the local machine
            with open(img_path, "wb") as f:
                f.write(img_response.content)
            print(f"Downloaded {img_name}")
        else:
            print(f"Failed to download image from {img_url}")
    except Exception as e:
        print(f"Error downloading {img_url}: {e}")

This script will download each image and save it to a folder named scraped_images in the current working directory. If the image URL is valid, it will be saved with the filename extracted from the URL.

Step 5: Save the Image URLs to a File (Optional)

If you want to save the URLs of the images for later reference, you can write them to a text file. Here’s an example of how to do this:


# Save image URLs to a text file
with open("image_urls.txt", "w") as file:
    for img_url in absolute_image_urls:
        file.write(img_url + "\n")

This will create a text file (image_urls.txt) and save each image URL on a new line.

✅ Pros of Scraping Images with Python

  • 🧠 Easy to Implement: With the help of BeautifulSoup and requests, scraping images is simple and straightforward.
  • ⚡ Efficient: You can scrape and download multiple images quickly with minimal lines of code.
  • 📂 Batch Download: Easily download and store large numbers of images with a single script.

⚠️ Cons of Scraping Images with Python

  • 💻 Legal Concerns: Always check the website’s robots.txt and terms of service to ensure that scraping images is allowed.
  • 🐢 Slow for Large Sites: If the site contains a lot of images or requires heavy processing, this method might take some time.
  • ❌ Non-Standard Formats: Some websites may use techniques like lazy loading or require you to bypass CAPTCHAs, which may complicate the process.

Summary

Scraping images with Python is a simple and powerful way to gather visual content from websites. By using BeautifulSoup to parse the HTML and requests to download the images, you can automate the process of collecting images from the web. This method works well for websites that expose images via standard HTML tags, though you may need to address issues like relative URLs or lazy loading for more complex sites.

For a much more extensive guide on web scraping with Python, including handling complex scenarios like authentication, pagination, and scraping at scale, check out my full 23-page PDF on Gumroad. It's available for just $10:

Mastering Web Scraping with Python Like a Pro.

If this was helpful, you can support me here: Buy Me a Coffee ☕