How To Scrape Websites With Python?

Web scraping with Python is the automated process of extracting data from websites using code rather than manual copying. It works by sending an HTTP request to a target website, retrieving its HTML structure, and parsing that HTML to extract specific data into structured formats like CSV or JSON.

Web scraping relies on a pipeline of standard web technologies. When you visit a page in a browser, your browser handles these steps automatically. When scraping, Python libraries replicate this behavior.

Core Libraries & Ecosystem

The Python scraping ecosystem is divided into three main categories based on the complexity of the target website:

Legal and Ethical Boundaries

Before writing a single line of code, you must understand the rules of the web:

• The robots.txt file: Always append /robots.txt to a website’s domain (e.g., ://example.com) to check which paths bots are allowed to crawl.
• Rate Limiting: Do not bombard a server with thousands of requests per second. Use delays (time.sleep) to mimic human behavior.
• Terms of Service (ToS): Some websites explicitly prohibit data scraping in their user agreements.
• Public vs. Private Data: Scrape only publicly available data. Avoid scraping data hidden behind login screens or containing Personal Identifiable Information (PII).

You can learn how to replace complex Excel macros with powerful Python scripts using a beginner’s guide.

Step by Step Guide for Beginners

This guide demonstrates how to scrape a static website using Requests and BeautifulSoup. We will target a safe sandbox site: ://toscrape.com.

Step 1: Set Up Your Environment

Create an isolated workspace and install the required tools using your terminal:

# Create and enter a project directory
mkdir python_scraper
cd python_scraper

# Install the necessary libraries
pip install requests beautifulsoup4

Step 2: Inspect Your Target Website

Before coding, you must find where your data lives in the source code.

1. Open your browser and go to the target page.
2. Right click on the element you want to scrape (e.g., a book title) and select Inspect (or press F12).
3. Note the HTML tags and classes. For instance, book containers might look like <article class=”product_pod”>, and titles might live inside <h3><a title=”Book Name”>…</a></h3>.

Step 3: Fetch the HTML Source Code

Use requests to download the web page content.

import requests

# Define the target URL
url = "https://toscrape.com"

# Send an HTTP GET request
response = requests.get(url)

# Check if the request was successful (Status Code 200)
if response.status_code == 200:
    print("Successfully connected to the website!")
    html_content = response.text
else:
    print(f"Failed to retrieve data. Status code: {response.status_code}")

Step 4: Parse HTML with BeautifulSoup

Convert the raw HTML string into a searchable tree structure.

from bs4 import BeautifulSoup

# Initialize BeautifulSoup with the HTML content and a parser
soup = BeautifulSoup(html_content, 'html.parser')

# Test by printing the website's title tag
print(soup.title.text.strip())

Step 5: Extract Specific Elements

Locate the container elements holding the data and extract the specific text or attributes.

# Find all book containers on the page
books = soup.find_all('article', class_='product_pod')

# Loop through each book to extract title and price
for book in books:
    # Extract the title attribute from the 'a' tag inside 'h3'
    title = book.h3.a['title']
    
    # Extract the price text from the paragraph with class 'price_color'
    price = book.find('p', class_='price_color').text
    
    print(f"Title: {title} | Price: {price}")

Step 6: Export Data to CSV

Save your scraped findings into a structured file format for analysis.

import csv

# Create and open a CSV file for writing
with open('books.csv', mode='w', newline='', encoding='utf-8') as file:
    writer = csv.writer(file)
    
    # Write the header row
    writer.writerow(['Title', 'Price'])
    
    # Write the extracted data rows
    for book in books:
        title = book.h3.a['title']
        price = book.find('p', class_='price_color').text
        writer.writerow([title, price])

print("Data successfully saved to books.csv!")

You can learn how to use SQL with step by step guide for beginners.

Handling Common Roadblocks

As you progress to real world websites, you will encounter security obstacles. Use these strategies to resolve them:

• HTTP 403 Forbidden Errors: Websites block requests missing a browser identity. Use a User-Agent header to make your request look like a standard web browser:

  headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'}
  response = requests.get(url, headers=headers)
  

IP Blocks and Captchas: If you scrape too aggressively, your IP will be flagged. Implement time.sleep (2) between requests, or integrate rotating proxy networks.

Missing Elements (Dynamic Content): If soup.find_all() returns nothing but you see the element in your browser, the page is rendered with JavaScript. You must switch to browser automation frameworks like Playwright to let the scripts execute before extracting the HTML.

For websites that require a login and multi page navigation (pagination), a basic requests.get() will not work. When you log in, the website server issues session cookies to your browser. Your Python script must capture these cookies and send them automatically with every subsequent request, otherwise, the server will log you out the moment you change pages.

What if the website requires logging in? How to scrape across multiple pages?

Here is a comprehensive guide to handling both login and pagination using static methods (requests.Session), along with strategies for modern JavaScript heavy websites.

If the login form exists directly inside the raw HTML (not loaded dynamically via JavaScript), Python’s requests.Session() is the perfect tool. It automatically manages cookies across all your requests.

Inspect the Login Form Payload

Before writing code, you need to know exactly what data the website expects when you click “Log In”.

1. Open your browser, right click the login page, and select Inspect (F12).
2. Go to the Network tab and check the Preserve Log box.
3. Log in manually with your account.
4. Look for a network request with a POST method (often named login, signin, or session).
5. Look at its Payload or Form Data tab. Note down the exact key names (e.g., user_email, pass_str, csrf_token).

Handling Hidden Security Tokens (CSRF)

Many websites include a hidden security input field inside the login form to prevent cross-site attacks. If your dictionary payload leaves this out, the server will block you with a 403 Forbidden error.

The Fix:
You must execute an initial GET request to fetch the login page structure, extract the token with BeautifulSoup, and dynamically insert it into your login dictionary.

# 1. Fetch the raw login page first
response = session.get(LOGIN_URL)
soup = BeautifulSoup(response.text, 'html.parser')

# 2. Extract the hidden token (names vary: 'csrf_token', 'authenticity_token', etc.)
csrf_token = soup.find('input', {'name': 'csrf_token'})['value']

# 3. Include it in your payload
login_payload = {
    'username': 'my_user',
    'password': 'my_password',
    'csrf_token': csrf_token  # Server will now validate your request
}

# 4. Fire the post login request
session.post(LOGIN_URL, data=login_payload)

What if the site uses JavaScript (React/Vue) or CAPTCHAs?

If the login form uses asynchronous validation or triggers a CAPTCHA challenge, the requests library will fail entirely because it cannot run browser engines.

For these modern, complex architectures, you must shift your strategy to a Headless Browser Automation tool like Playwright or Selenium.

Instead of dealing manually with raw headers and session objects, you write explicit browser automation scripts that mimic actual human behavior:

1. page.goto(‘https://example.com’) opens a visible or invisible automated Chromium window.
2. page.fill(‘#username-input’, ‘my_user’) types into the fields.
3. page.click(‘#submit-btn’) executes the JavaScript logins natively.
4. Pagination is accomplished by target clicking the actual “Next” button element (page.click(‘text=Next Page’)) rather than changing numerical query strings in URLs.