How to Build a Web Scraper with Python in 2026: Complete Ethical Guide

Web scraping extracts data from websites programmatically — useful for research, price monitoring, data aggregation, and automation. This guide builds scrapers in Python, from simple static pages to dynamic JavaScript sites, while covering the legal and ethical practices that keep you out of trouble.

Scrape Responsibly First

Before writing any scraper, understand the rules:

• Check robots.txt: example.com/robots.txt tells you what’s allowed to be crawled
• Read the Terms of Service: Some sites explicitly prohibit scraping
• Respect rate limits: Don’t hammer a server — add delays between requests
• Use official APIs when available: Always prefer an API over scraping
• Don’t scrape personal data without a legal basis (GDPR and similar laws apply)
• Identify yourself: Set a proper User-Agent, don’t disguise your bot maliciously

Setup

pip install requests beautifulsoup4 lxml
# For JavaScript-heavy sites:
pip install playwright
playwright install chromium

Scraping Static Pages with requests + BeautifulSoup

import requests
from bs4 import BeautifulSoup
import time

def scrape_articles(url):
    headers = {
        'User-Agent': 'Mozilla/5.0 (compatible; MyResearchBot/1.0)'
    }
    response = requests.get(url, headers=headers, timeout=10)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, 'lxml')

    articles = []
    for article in soup.select('article.post'):
        title = article.select_one('h2.title')
        link  = article.select_one('a')
        articles.append({
            'title': title.get_text(strip=True) if title else None,
            'url':   link['href'] if link else None,
        })
    return articles

data = scrape_articles('https://example.com/blog')
for item in data:
    print(item)

Selecting Elements Precisely

soup = BeautifulSoup(html, 'lxml')

# By tag
soup.find('h1')                        # first h1
soup.find_all('p')                     # all paragraphs

# By CSS selector (most flexible)
soup.select('div.card > h2')           # h2 inside div.card
soup.select_one('#main .price')        # first .price inside #main

# Extract data
element.get_text(strip=True)           # text content
element['href']                        # attribute value
element.get('data-id', 'default')      # attribute with fallback

# Navigate
element.parent                         # parent element
element.find_next_sibling('div')       # next sibling

Scraping Dynamic (JavaScript) Sites with Playwright

Many modern sites render content with JavaScript — requests won’t see it. Playwright runs a real browser:

from playwright.sync_api import sync_playwright

def scrape_dynamic(url):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(user_agent='Mozilla/5.0 (MyBot/1.0)')

        page.goto(url, wait_until='networkidle')

        # Wait for content to load
        page.wait_for_selector('.product-card')

        # Extract data from the rendered page
        products = page.eval_on_selector_all('.product-card', '''
            cards => cards.map(card => ({
                name: card.querySelector('.name')?.innerText,
                price: card.querySelector('.price')?.innerText,
            }))
        ''')

        browser.close()
        return products

results = scrape_dynamic('https://example.com/products')

Rate Limiting and Politeness

import time
import random

def polite_scrape(urls):
    results = []
    for url in urls:
        try:
            data = scrape_page(url)
            results.append(data)
        except requests.RequestException as e:
            print(f"Failed {url}: {e}")

        # Random delay between requests (1-3 seconds) - be gentle
        time.sleep(random.uniform(1, 3))

    return results

# Respect Retry-After on 429 responses
response = requests.get(url)
if response.status_code == 429:
    retry_after = int(response.headers.get('Retry-After', 60))
    time.sleep(retry_after)

Handling Pagination

def scrape_all_pages(base_url):
    all_data = []
    page = 1

    while True:
        url = f"{base_url}?page={page}"
        response = requests.get(url, headers=HEADERS, timeout=10)
        soup = BeautifulSoup(response.text, 'lxml')

        items = soup.select('.item')
        if not items:
            break   # no more items - stop

        all_data.extend(extract_items(items))

        # Check for a "next" link, or just increment
        if not soup.select_one('a.next-page'):
            break
        page += 1
        time.sleep(random.uniform(1, 2))

    return all_data

Saving Scraped Data

import csv
import json

# To CSV
with open('data.csv', 'w', newline='') as f:
    writer = csv.DictWriter(f, fieldnames=['title', 'url', 'price'])
    writer.writeheader()
    writer.writerows(data)

# To JSON
with open('data.json', 'w') as f:
    json.dump(data, f, indent=2, ensure_ascii=False)

# To a database (SQLite example)
import sqlite3
conn = sqlite3.connect('scraped.db')
conn.executemany(
    "INSERT INTO products (name, price) VALUES (?, ?)",
    [(d['name'], d['price']) for d in data]
)
conn.commit()

Frequently Asked Questions

Q: Is web scraping legal?
A: It depends. Scraping publicly available data is often legal, but violating Terms of Service, scraping personal data without basis, or bypassing access controls can be illegal. Check robots.txt and ToS, prefer official APIs, and consult legal advice for commercial scraping.

Q: requests/BeautifulSoup or Playwright?
A: Use requests + BeautifulSoup for static HTML pages (faster, lighter). Use Playwright (or Selenium) for JavaScript-rendered sites where content loads dynamically. Check if the data is in the initial HTML first — if so, you don’t need a browser.

Q: How do I avoid getting blocked?
A: Scrape politely — respect rate limits, add delays, set a proper User-Agent, and don’t overload servers. The best way to avoid blocks is to not be abusive. Aggressive scraping gets blocked (and may be illegal).

Q: Should I use an API instead of scraping?
A: Always, if one exists. APIs are more reliable, legal, structured, and don’t break when the site’s HTML changes. Scraping is a last resort when no API provides the data you need.

Q: How do I handle sites that require login?
A: Playwright can automate login (fill forms, submit), maintaining a session. But scraping behind a login often violates Terms of Service — be especially careful about legality and only scrape data you’re authorized to access.

Conclusion

Web scraping with Python is powerful for data collection, but do it responsibly. Use requests + BeautifulSoup for static pages and Playwright for JavaScript-rendered sites, select elements with CSS selectors, handle pagination, and save to CSV/JSON/database. Most importantly: check robots.txt and Terms of Service, respect rate limits with delays, prefer official APIs, and don’t scrape personal data without a legal basis. Ethical, polite scraping keeps you out of legal and technical trouble while getting the data you need. When an API exists, always use it instead — scraping is the fallback, not the first choice.