Web scraping is the automated collection of data from websites. Python web scraping means using Python code to visit web pages and pull out the data you need, such as prices, headlines or table rows. In this guide, we'll dive deep into the nitty-gritty of web scraping in Python.
We’ll build a working scraper with Requests and Beautiful Soup. Then, we'll show which tool to reach for when a page needs JavaScript, blocks you, or has thousands of URLs.
I ran all the code on Python 3.15 and benchmarked 6 HTML parsers. I also measured 6 ways of fetching the same 8 public pages: 6 practice pages built for scraping, plus Hacker News and a Wikipedia page. You can rerun everything from the scripts, URL list and raw results in the public GitHub repo.

Key takeaways:
- Start with Requests and Beautiful Soup. Requests got the data on 439 of 440 requests to six static pages in my test.
- On the two JavaScript pages, Requests, httpx and curl_cffi got zero quotes. Playwright got them on all 200 requests; ScrapingBee's Auto Mode on 199.
- selectolax parsed a 2 MB Wikipedia page in 8 ms. Beautiful Soup took 172 ms with html.parser and still 130 ms with lxml.
- A browser User-Agent doesn't change your TLS fingerprint. curl_cffi changes it, and a scraping API handles it for you.
- Scrape public pages only, read robots.txt, slow down, and stop when you see repeated 403 or 429 responses.
Getting started with Python web scraping
Python is a popular choice for web scraping because every step, from downloading to parsing to saving, has a mature library, and the same script can clean and store the results. For instance, a program can request a page, read the HTML that comes back and keep only the parts you need, such as a price, a title or a table row.
If you're new to web scraping, our guide to what web scraping is covers its uses and how it works before you write any code.
Many scraping projects I’ve seen go wrong before the first line of code. Someone reaches for a headless browser (a browser that runs without a window) to read a page that a plain HTTP request would have handled in under half a second.
So before we write a scraper, we'll install the tools and match each kind of page to the smallest one that works.
Installing Python and the scraping libraries
First, you need Python 3.12 or newer. Python 3.9 reached end of life (no more security fixes) in October 2025, and 3.10 followed on October 1, 2026. Requests has required Python 3.10 or later since version 2.33.

For this guide, I ran everything on Python 3.15.2, the latest release as of this writing, and again on 3.14.10, where every snippet worked unchanged.
I use uv to manage the project because it installs Python, creates the virtual environment and locks package versions, all in one tool. A virtual environment is an isolated folder of packages for one project.
Create a project and add the three libraries the first scraper needs:
uv init --no-package scraper
cd scraper
uv add requests beautifulsoup4 lxml
uv run main.py
The --no-package flag matters: since uv 0.12, a bare uv init sets up an installable package with a src/ folder, more structure than a scraping script needs. With the flag, you get a single main.py to run with uv run.
If you'd rather stay with the tools that ship with Python, this is the venv and pip equivalent:
python3 -m venv .venv
source .venv/bin/activate # Windows PowerShell: .venv\Scripts\Activate.ps1
pip install requests beautifulsoup4 lxml
These are the packages this tutorial uses, with the versions I tested on November 1, 2026 (printed with uv pip list):
| Package | Tested version | What it does here |
|---|---|---|
| requests | 2.34.2 | Downloads pages over HTTP |
| beautifulsoup4 | 4.15.0 | Finds data in HTML with CSS selectors |
| lxml | 6.1.3 | Fast HTML parser for Beautiful Soup, plus XPath |
| curl_cffi | 0.17.0 | HTTP client that looks like Chrome at the TLS level |
| httpx | 0.28.1 | HTTP client with async support |
| playwright | 1.64.1 | Drives a real browser for JavaScript pages |
| selenium | 4.52.0 | The older browser automation option |
| scrapy | 2.19.0 | Crawls whole sites |
| scrapingbee | 2.2.0 | ScrapingBee's Python SDK |
To install everything at once:
uv add curl_cffi httpx playwright selenium scrapy scrapingbee pandas
Matching each kind of page to a tool
The right tool depends on where the data lives and how the site responds to a plain request:
- Use Requests and Beautiful Soup when the data is in the page's HTML.
- Switch to curl_cffi when a public page blocks plain Requests.
- Use Playwright when the data appears only after JavaScript runs.
- Use Scrapy when you need to crawl thousands of pages.
- Use a web scraping API when you need JavaScript rendering and fewer blocks without running your own browsers or proxies (servers that forward your requests from other IP addresses).

This table is the same decision in text form, with the cost of each choice:
| Page type | Tool | Why | What it costs you |
|---|---|---|---|
| Static HTML (the data is in View Source) | Requests and Beautiful Soup | Smallest setup, fast on static pages | Minutes to set up; a median of 0.40 s per page in my test |
| A public page that blocks plain Requests | curl_cffi | Sends a Chrome TLS and HTTP/2 fingerprint | One import change; no help against JavaScript challenges |
| A JavaScript-rendered page | Playwright | Runs the page's JavaScript in a real browser | About 570 MB of browser files on disk and about 10x the memory of Requests |
| Many pages or a whole site | Scrapy | Scheduling, retries, throttling and exports built in | An hour to learn the project layout |
| Many pages at once from your own code | asyncio with httpx | Keeps several requests in flight | More code; easy to overload a small site |
| Protected or JavaScript-heavy pages at volume | A web scraping API | Rendering and proxies run on someone else's servers | Credits per page (1 to 75 on ScrapingBee) |
Checking the Network tab for a JSON endpoint first
Pages that look dynamic often load their data from a JSON endpoint (a URL that returns raw data instead of HTML). Requesting that endpoint is faster and more stable than parsing HTML or running a browser.
Before I write a scraper for any page, I spend a few minutes in the browser's developer tools looking for one.
Here's the routine on quotes.toscrape.com/scroll, a practice page that loads quotes as you scroll:
- Open the page in Chrome, press F12 (Cmd+Option+I on a Mac) and open the Network tab.
- Click the Fetch/XHR filter and scroll the page.
- Watch for requests that return JSON. Here it's api/quotes?page=2, page=3 and so on.
- Right-click the request, choose Copy > Copy as cURL, and paste it into a cURL-to-Python converter or read the URL and headers from it.
The endpoint returns 10 quotes per page plus a has_next flag, so a loop with Requests collects everything without touching HTML:
import time
import requests
API_URL = "https://quotes.toscrape.com/api/quotes"
HEADERS = {"User-Agent": "my-first-scraper/1.0 (+https://example.com/contact)"}
quotes, page = [], 1
with requests.Session() as session:
session.headers.update(HEADERS)
while True:
response = session.get(API_URL, params={"page": page}, timeout=10)
response.raise_for_status()
data = response.json()
quotes.extend(data["quotes"])
if not data["has_next"]:
break
page += 1
time.sleep(1) # one request per second, even on an API
print(f"{len(quotes)} quotes from {page} JSON pages")
print(quotes[0]["text"], "by", quotes[0]["author"]["name"])
The loop made ten requests, collected 100 quotes and needed no parsing:
100 quotes from 10 JSON pages
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” by Albert Einstein
Two more places can hold ready-made structured data. Product and article pages often embed a <script type="application/ld+json"> block with the name, price and rating.
Sites built with Next.js (a popular JavaScript framework) put their page data in a <script id="__NEXT_DATA__"> tag. Search View Source (Ctrl+U) for either one before you write a single selector. If the site also offers an official API, use that first.
How to web scrape with Python using Requests and Beautiful Soup
Requests downloads the page, and Beautiful Soup turns the HTML into something you can search. You then pick out the fields with CSS selectors (patterns that match HTML elements by tag, class or ID).
We'll scrape the Hacker News front page first. Then we'll follow the pagination links on Books to Scrape and save the results.
1. Inspect the page in DevTools
Open Hacker News in Chrome, right-click a story title and choose Inspect.

The Elements panel jumps to the HTML behind that title.

Three patterns are enough for the scraper:
- Each story is a table row with the classes athing submission, so tr.athing.submission selects all 30 of them.
- The title link sits inside .titleline > a.
- The row after each story holds the points, the author, and the comment count inside .subtext.
Before you trust a selector, press Ctrl+F (Cmd+F on a Mac) in the Elements panel and paste it in. DevTools shows how many elements match, so you know tr.athing.submission finds 30 rows before you've written any Python.
2. Fetch pages with a Session, timeouts and an honest User-Agent
Set up these three things before your first request:
- Use a requests.Session. It reuses the connection between requests and keeps your headers in one place.
- Pass a timeout on every request. Requests has no default timeout, so without one, a stalled server can freeze your script forever. timeout=(5, 15) allows 5 seconds to connect, then gives up if the server sends nothing for 15 seconds.
- Say who you are in the User-Agent. The default User-Agent (the header that names your client) is python-requests/2.34.2, which tells the site nothing. A name plus a URL or email where the site owner can reach you is polite and, on sites like Wikipedia, required by policy.
The scraper in the next step uses all three, and its fetch_html() function is a good starting point for your own projects. Then format fetch_html() as code.
3. Parse the data into a dataclass
Beautiful Soup is a Python library that reads HTML and lets you search it by tag, class or CSS selector. Save this scraper as hn_scraper.py and run it with uv run hn_scraper.py (or python hn_scraper.py in an activated venv):
from __future__ import annotations
import re
from dataclasses import dataclass
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
HN_URL = "https://news.ycombinator.com/"
HEADERS = {"User-Agent": "my-first-scraper/1.0 (+https://example.com/contact)"}
TIMEOUT = (5, 15) # 5 s to connect, then give up after 15 s with no data
@dataclass
class Story:
rank: int
title: str
url: str
points: int
author: str
comments: int
def extract_number(text: str | None) -> int:
"""Return the first whole number in a string like '31 points', or 0."""
if not text:
return 0
match = re.search(r"\d+", text.replace("\xa0", " "))
return int(match.group()) if match else 0
def fetch_html(session: requests.Session, url: str) -> str:
response = session.get(url, timeout=TIMEOUT)
response.raise_for_status()
return response.text
def parse_stories(html: str, limit: int = 5) -> list[Story]:
soup = BeautifulSoup(html, "lxml")
stories = []
for row in soup.select("tr.athing.submission")[:limit]:
title_link = row.select_one(".titleline > a")
if title_link is None: # layout changed or a special row: skip it
continue
subtext = row.find_next_sibling("tr")
score = subtext.select_one(".score") if subtext else None
author = subtext.select_one(".hnuser") if subtext else None
comment_links = subtext.find_all("a", string=re.compile("comment")) if subtext else []
rank = row.select_one(".rank")
stories.append(Story(
rank=extract_number(rank.get_text() if rank else None),
title=title_link.get_text(strip=True),
url=urljoin(HN_URL, title_link["href"]),
points=extract_number(score.get_text() if score else None),
author=author.get_text(strip=True) if author else "unknown",
comments=extract_number(comment_links[0].get_text() if comment_links else None),
))
return stories
def main() -> None:
with requests.Session() as session:
session.headers.update(HEADERS)
html = fetch_html(session, HN_URL)
for story in parse_stories(html):
print(f"{story.rank}. {story.title}")
print(f" {story.url}")
print(f" {story.points} points by {story.author}, {story.comments} comments")
if __name__ == "__main__":
main()
It fetches the front page once and turns the first five stories into Story objects, defined with a dataclass (a short way to define a class that holds data).
Running it on November 1, 2026 printed this (your stories will differ, since the front page changes all day):
1. Show HN: HackerBooks – HackerTube, but for books mentioned on HN
https://hackerbooks.org
275 points by roansh, 87 comments
2. HackerTube: YouTube built from videos recommended by HN
https://karpathy.bearblog.dev/hacker-tube/
1857 points by karpathy, 477 comments
3. Seoul Viosys begins production of rare-earth-free white light emitter
https://www.semiconductor-today.com/news_items/2026/oct/seoul-291026.shtml
74 points by gmays, 9 comments
4. Police chief unplugs Flock cameras after company ignores contract cancellation
https://www.tomshardware.com/tech-industry/big-tech/police-chief-physically-unplugs-flock-cameras-after-the-company-ignores-contract-cancellation-city-takes-matters-into-its-own-hands-amid-privacy-controversies
248 points by entryleveljobsc, 72 comments
5. Patrick Stewart taught us what good men do with power
https://www.vox.com/politics/508526/patrick-stewart-jean-luc-picard-star-trek-liberalism
58 points by rhaynes, 41 comments
Four Beautiful Soup conventions in that scraper are worth copying:
(1) select() and select_one() take CSS selectors, the same ones you tested in DevTools. select() returns a list and select_one() returns the first match or None.
(2) get_text(strip=True) returns clean text without the surrounding whitespace. HN separates numbers from words with a non-breaking space (\xa0), which is why extract_number() replaces it before matching digits.
(3) Use the snake_case names: find_all, never findAll. Beautiful Soup 4.13 started warning about the old camelCase names, and the changelog calls 4.15 the last release that supports them.
A later point release (a minor update) will make the old names raise NotImplementedError, and after that the maintainers plan to remove them. From then on, Beautiful Soup reads findAll as a search for a <findAll> tag. So soup.findAll("a") will fail with a confusing TypeError: 'NoneType' object is not callable.
(4) Check for None before you call a method on a result. None.get_text() crashes the whole run with an AttributeError. The if title_link is None: continue line turns a layout change into a skipped row instead.
The "lxml" argument picks the parser, the component that turns raw HTML into a tree of elements you can search. lxml is a C library, so it builds the HTML tree faster than Python's built-in html.parser. The parser benchmark has the numbers.
4. Check robots.txt before you loop
robots.txt is a plain text file at the root of a site. It tells crawlers (programs that visit pages automatically) which paths the owner would rather they skip and, sometimes, how long to wait between requests.
Python ships a parser for it in urllib.robotparser, but its own read() method has a trap I hit in my tests. read() downloads robots.txt with urllib's default User-Agent, which some sites, including Wikipedia and scrapethissite.com, reject with a 403 (Forbidden) status. robotparser treats a 403 as "everything is disallowed", so can_fetch() said no to pages that were allowed.
The fix is to download robots.txt with your own Session and hand the text to parse(). The comments in the helper show how it handles each status code:
from urllib.robotparser import RobotFileParser
import requests
HEADERS = {"User-Agent": "my-first-scraper/1.0 (+https://example.com/contact)"}
def load_robots(session: requests.Session, site: str) -> RobotFileParser:
parser = RobotFileParser()
response = session.get(f"{site}/robots.txt", timeout=10)
if response.status_code in (404, 410):
parser.parse([]) # no robots.txt: RFC 9309 says everything is allowed
elif response.status_code in (401, 403):
parser.parse(["User-agent: *", "Disallow: /"]) # refused: treat the site as off-limits
else:
response.raise_for_status() # 429 or 5xx: back off and try again later
parser.parse(response.text.splitlines())
return parser
with requests.Session() as session:
session.headers.update(HEADERS)
for site, path in [("https://news.ycombinator.com", "/"),
("https://news.ycombinator.com", "/login"),
("https://books.toscrape.com", "/catalogue/page-2.html")]:
robots = load_robots(session, site)
agent = HEADERS["User-Agent"]
if robots.can_fetch(agent, site + path):
delay = robots.crawl_delay(agent)
print(f"OK to fetch {site}{path}" + (f", waiting {delay} s between requests" if delay else ""))
else:
print(f"Skipping {site}{path}: robots.txt doesn't allow it")
The output says which paths you may fetch, which to skip and how long to wait:
OK to fetch https://news.ycombinator.com/, waiting 30 s between requests
Skipping https://news.ycombinator.com/login: robots.txt doesn't allow it
OK to fetch https://books.toscrape.com/catalogue/page-2.html
Hacker News asks for 30 seconds between requests, so a scraper that loops over its pages once a second ignores the owner's stated wishes. Books to Scrape has no robots.txt at all. Under RFC 9309, the standard that defines the file, a missing robots.txt means no restrictions.
5. Follow pagination
Most listings span many pages, and the reliable way through them is to follow the site's own "next" link rather than guess URLs.
This scraper collects every book on Books to Scrape, follows li.next a until there isn't one, and waits a second between pages:
import time
from datetime import datetime, timezone
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
START_URL = "https://books.toscrape.com/"
HEADERS = {"User-Agent": "my-first-scraper/1.0 (+https://example.com/contact)"}
def scrape_books(max_pages: int = 50) -> list[dict]:
books, url, pages = [], START_URL, 0
with requests.Session() as session:
session.headers.update(HEADERS)
while url and pages < max_pages:
response = session.get(url, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.content, "lxml") # bytes: let Beautiful Soup read the page's charset
scraped_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
for card in soup.select("article.product_pod"):
link = card.select_one("h3 a")
price = card.select_one(".price_color")
if link is None or price is None:
continue
books.append({
"title": link["title"],
"price": price.get_text(strip=True),
"url": urljoin(url, link["href"]),
"source_url": url,
"scraped_at": scraped_at,
})
pages += 1
next_link = soup.select_one("li.next a")
url = urljoin(url, next_link["href"]) if next_link else None
if url:
time.sleep(1) # one request per second is plenty for a small site
print(f"Scraped {len(books)} books from {pages} pages")
return books
if name == "__main__":
start = time.perf_counter()
books = scrape_books()
print(books[0])
print(f"Took {time.perf_counter() - start:.1f} s")
All 50 pages took 74.7 seconds, most of it the deliberate one-second pause:
Scraped 1000 books from 50 pages
{'title': 'A Light in the Attic', 'price': '£51.77', 'url': 'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html', 'source_url': 'https://books.toscrape.com/', 'scraped_at': '2026-11-01T18:05:02+00:00'}
Took 74.7 s
The first time I ran this loop, the price printed as £51.77, not £51.77. Books to Scrape sends its pages with a bare Content-Type: text/html header and no character set (the encoding that maps bytes to letters).
Requests then falls back to ISO-8859-1 when it builds response.text. As a result, the UTF-8 pound sign comes out mangled. Passing response.content, the raw bytes, to Beautiful Soup fixes the price, because Beautiful Soup then reads the <meta charset> tag inside the page.
Hacker News declares charset=utf-8 in its header, which is why the first scraper didn't need the fix. Make response.content your default with Beautiful Soup.
urljoin(url, link["href"]) turns relative links (links that leave out the domain) such as catalogue/page-2.html into full URLs. It resolves each link against the page it came from, which matters here because the first page and the pages under /catalogue/ use different relative paths.
6. Save the results with a source URL and a timestamp
Every record in the pagination scraper already carries source_url and scraped_at. Keep both on everything you scrape. When a number looks wrong later on, they tell you where it came from and how old it is.
To save the books three ways, add these imports to the top of the pagination script and replace its if name == "main": block with the rest:
import csv
import json
import sqlite3
books = scrape_books(max_pages=2)
# CSV: opens in any spreadsheet
with open("books.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=books[0].keys())
writer.writeheader()
writer.writerows(books)
# JSON Lines: one record per line, safe to append on every run
with open("books.jsonl", "a", encoding="utf-8") as f:
for book in books:
f.write(json.dumps(book, ensure_ascii=False) + "\n")
# SQLite: query, dedupe and update over time
with sqlite3.connect("books.db") as db:
db.execute("""CREATE TABLE IF NOT EXISTS books (
url TEXT PRIMARY KEY, title TEXT, price TEXT, source_url TEXT, scraped_at TEXT)""")
db.executemany(
"INSERT OR REPLACE INTO books VALUES (:url, :title, :price, :source_url, :scraped_at)", books)
print(db.execute("SELECT COUNT(*) FROM books").fetchone()[0], "books in books.db")
The run fetched two pages and saved 40 books to three files:
Scraped 40 books from 2 pages
40 books in books.db
The first line of books.jsonl shows the shape every record keeps:
{"title": "A Light in the Attic", "price": "£51.77", "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "source_url": "https://books.toscrape.com/", "scraped_at": "2026-11-01T18:07:06+00:00"}
Pick the format by what happens next:
- CSV opens in Excel or Google Sheets. Write it with newline="" so Windows doesn't insert blank rows.
- JSON Lines writes one JSON object per line so that you can read huge files one record at a time. Use it for anything you'll add to over time.
- SQLite comes with Python and gives you queries, updates and deduplication. Here the URL is the primary key (the column that must be unique), so rerunning the scraper updates rows instead of duplicating them.
When the data is already an HTML table, pandas.read_html skips the selectors. It turns every <table> on a page into a DataFrame, pandas' table object (install pandas with uv add pandas):
from io import StringIO
import pandas as pd
import requests
HEADERS = {"User-Agent": "my-first-scraper/1.0 (+https://example.com/contact)"}
response = requests.get("https://www.scrapethissite.com/pages/forms/", headers=HEADERS, timeout=10)
response.raise_for_status()
teams = pd.read_html(StringIO(response.text))[0]
print(teams.shape)
print(teams[["Team Name", "Year", "Wins", "Losses"]].head(3))
The hockey table on scrapethissite.com came back as a ready-made DataFrame:
(25, 9)
Team Name Year Wins Losses
0 Boston Bruins 1990 44 24
1 Buffalo Sabres 1990 31 30
2 Calgary Flames 1990 46 26
I fetch the page with Requests instead of passing the URL to read_html. pandas would download it with the same urllib User-Agent that tripped up robotparser. scrapethissite.com answers that User-Agent with a 403.
For more on table scraping, see our guide to scraping HTML tables.
Rendering dynamic pages in Python with Playwright
Some pages arrive almost empty and build their content in the browser with JavaScript. Requests only downloads what the server sends, so on those pages it gets the empty shell. If you can see the data in the browser but not in View Source, the page is dynamic.
Why Requests sees an empty page
quotes.toscrape.com/js/ looks identical to the static quotes page in a browser.
Here's what Requests gets from it:
import requests
from bs4 import BeautifulSoup
HEADERS = {"User-Agent": "my-first-scraper/1.0 (+https://example.com/contact)"}
response = requests.get("https://quotes.toscrape.com/js/", headers=HEADERS, timeout=10)
soup = BeautifulSoup(response.text, "lxml")
print("Status:", response.status_code)
print("Quotes found:", len(soup.select("div.quote")))
print("Quote text sits in a script:", "var data" in response.text)
A 200 (OK) status and zero quotes:
Status: 200
Quotes found: 0
Quote text sits in a script: True
That's the trap with dynamic pages: nothing fails, and you get an empty list. The quotes are inside a <script> tag as a JavaScript array that only becomes HTML when a browser runs it.
Before you switch to a browser, look at that script or the Network tab. If the data sits in JSON, you can read it without rendering anything.
Scraping with Playwright
Playwright drives a Chromium, Firefox or WebKit browser from Python, so the page's JavaScript runs exactly as it would for a visitor.
Install it and download the browser it controls:
uv add playwright
uv run playwright install chromium
On Linux, use uv run playwright install --with-deps chromium so it also installs the system libraries the browser needs.
This scraper targets the delayed version of the page, quotes.toscrape.com/js-delayed/, which waits 10 seconds before it renders anything:
from future import annotations
from dataclasses import dataclass
from playwright.sync_api import TimeoutError as PlaywrightTimeoutError
from playwright.sync_api import sync_playwright
URL = "https://quotes.toscrape.com/js-delayed/" # renders quotes after a 10-second delay
WAIT_MS = 20_000
@dataclass
class Quote:
text: str
author: str
tags: list[str]
def scrape_quotes(limit: int = 3) -> list[Quote]:
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
try:
page = browser.new_page()
page.goto(URL, wait_until="domcontentloaded")
cards = page.locator("div.quote")
cards.first.wait_for(timeout=WAIT_MS) # waits for the element, not a guessed sleep
quotes = []
for card in cards.all()[:limit]:
quotes.append(Quote(
text=card.locator(".text").inner_text(),
author=card.locator(".author").inner_text(),
tags=card.locator(".tag").all_inner_texts(),
))
return quotes
except PlaywrightTimeoutError as exc:
raise RuntimeError(f"No quotes after {WAIT_MS / 1000:.0f} s") from exc
finally:
browser.close()
if name == "__main__":
for quote in scrape_quotes():
print(quote.text)
print(f" {quote.author} | {', '.join(quote.tags)}")
The script printed the first three quotes after the delay:
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
Albert Einstein | change, deep-thoughts, thinking, world
“It is our choices, Harry, that show what we truly are, far more than our abilities.”
J.K. Rowling | abilities, choices
“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”
Albert Einstein | inspirational, life, live, miracle, miracles
The script makes three choices:
(1) cards.first.wait_for(timeout=WAIT_MS) waits for the element itself. Playwright now discourages page.wait_for_selector() in favor of locators (objects that find elements and wait for them).
A locator wait is better than time.sleep() because it returns the moment the quotes appear. Locator actions such as inner_text() also wait on their own.
(2) The timeout is 20 seconds, not the 30-second default or a short 5-second one. This page delays rendering by 10 seconds, so a tight timeout fails every time. If you're only testing, ?delay=2000 on the URL shortens the delay to 2 seconds.
(3) try and finally close the browser even when the wait times out, so a failed run doesn't leave Chromium processes using memory in the background.
I ran this on Playwright 1.64.1.
When a browser is worth the extra time and memory
A browser is the right tool when the data only exists after the page runs code and you can't find a JSON endpoint behind it. Typical cases are infinite scroll, "Load more" buttons, search results built in the browser and dashboards that fetch their numbers after the page opens.
In my tests, the one-page Playwright scraper peaked at 502.5 MB of memory across its browser processes, against 48.5 MB for the same page with Requests. A static page took a median of 1.63 s with Playwright against 0.40 s with Requests.
The difference barely matters for 50 pages, but it decides your server bill at 50,000. Our guide to scraping dynamic pages has more patterns for sites built with JavaScript.
Where Selenium still fits
If your team already has Selenium scrapers, they still work, and Selenium 4 is far easier to set up. Install it with uv add selenium. It drives the Google Chrome you have installed.
Selenium Manager, built into the library, downloads a matching driver (the small program that lets Selenium control Chrome), so there's no chromedriver to install by hand. You find elements with find_element(By.CSS_SELECTOR, ...).
Selenium 4.3.0 removed the old find_element_by_css_selector() style, so 2020-era scripts fail with an AttributeError on the first lookup. The same /js/ quotes scrape in Selenium 4 looks like this:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options) # Selenium Manager fetches a matching driver
try:
driver.get("https://quotes.toscrape.com/js/")
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "div.quote")))
for card in driver.find_elements(By.CSS_SELECTOR, "div.quote")[:3]:
text = card.find_element(By.CSS_SELECTOR, ".text").text
author = card.find_element(By.CSS_SELECTOR, ".author").text
print(f"{text} ({author})")
finally:
driver.quit()
Selenium 4.52.0 found my installed Chrome 156, fetched the matching driver and printed:
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” (Albert Einstein)
“It is our choices, Harry, that show what we truly are, far more than our abilities.” (J.K. Rowling)
“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.” (Albert Einstein)
For new scrapers, I pick Playwright, because it waits for elements on its own and installs its browsers with one command.
How to scale Python web scraping to thousands of pages
A loop with Requests is fine for a few hundred pages. Past that, you start rebuilding things that already exist (a URL queue, retries, politeness delays, deduplication and export). Scrapy gives you all of it as a framework, and asyncio lets you keep your own code while sending several requests at once.
1. Use Scrapy to crawl a whole site
Scrapy is a crawling framework. You write the parsing logic, and it handles scheduling, retries, throttling, robots.txt and exporting. Here, we use version 2.19.0, released on September 10, 2026.
To start, create a project and a spider (a class that tells Scrapy which pages to visit and how to parse them):
uv add scrapy
uv run scrapy startproject bookcrawler
cd bookcrawler
uv run scrapy genspider books books.toscrape.com
New projects start polite. The generated settings.py already sets:
ROBOTSTXT_OBEY = True
CONCURRENT_REQUESTS_PER_DOMAIN = 1
DOWNLOAD_DELAY = 1
Those settings mean one request at a time per site, a second apart, with robots.txt respected, which is the right default for a site you don't own. The one thing to add is an honest User-Agent, since Scrapy otherwise identifies itself only as Scrapy/2.19.0 (+https://scrapy.org).
Find the commented-out USER_AGENT line in settings.py and set it:
USER_AGENT = "bookcrawler (+https://example.com/contact)"
Raise CONCURRENT_REQUESTS_PER_DOMAIN and lower DOWNLOAD_DELAY only on your own servers, or after checking that the site accepts more traffic. Scrapy spreads the delay randomly between 0.5x and 1.5x so requests don't arrive at perfectly regular intervals.
Since 2.19, the setting for that is DOWNLOAD_DELAY_JITTER, which replaced the older RANDOMIZE_DOWNLOAD_DELAY.
Writing and running the spider
Replace bookcrawler/spiders/books.py with this spider. It walks two listing pages, follows every book to its detail page, and passes the listing's star rating to the detail callback (the method that parses each book page):
import scrapy
class BooksSpider(scrapy.Spider):
name = "books"
allowed_domains = ["books.toscrape.com"]
start_urls = ["https://books.toscrape.com/"]
max_pages = 2 # keep the demo crawl small
custom_settings = {
"FEEDS": {"books.jsonl": {"format": "jsonlines", "overwrite": True}},
}
def parse(self, response, page=1):
for card in response.css("article.product_pod"):
yield response.follow(
card.css("h3 a::attr(href)").get(),
callback=self.parse_book,
cb_kwargs={"rating": card.css("p.star-rating::attr(class)").re_first(r"star-rating (\w+)"), "page": page},
)
next_page = response.css("li.next a::attr(href)").get()
if next_page and page < self.max_pages:
yield response.follow(next_page, callback=self.parse, cb_kwargs={"page": page + 1})
def parse_book(self, response, rating, page):
yield {
"title": response.css("div.product_main h1::text").get(),
"price": response.css("p.price_color::text").get(),
"in_stock": response.css("p.availability::text").re_first(r"\d+"),
"upc": response.css("th:contains('UPC') + td::text").get(),
"rating": rating,
"listing_page": page,
"url": response.url,
}
Run it from the project folder:
uv run scrapy crawl books
The crawl made 43 requests (robots.txt, two listing pages and 40 book pages) and saved 40 records. It took 51 seconds because the one-second delay applies to every request. This is the first record it saved:
{
"title": "It's Only the Himalayas",
"price": "£45.17",
"in_stock": "19",
"upc": "a22124811bfa8350",
"rating": "Two",
"listing_page": 1,
"url": "https://books.toscrape.com/catalogue/its-only-the-himalayas_981/index.html"
}
Four details mark an up-to-date Scrapy project in 2026:
(1) start_urls and start() set the first requests. For custom first requests, such as adding headers or reading URLs from a file, define async def start(self) and yield requests from it. start() replaced start_requests(), which Scrapy deprecated in 2.13 and removed in 2.16.0 (release notes).
When I ran a 2020-style spider that only defines start_requests() on Scrapy 2.19, it exited cleanly with "Crawled 0 pages" and no warning. If an old tutorial's spider scrapes nothing, look for start_requests() first.
(2) cb_kwargs passes values between callbacks. Older code stuffs values into response.meta, which still works but mixes your data with Scrapy's internal keys.
(3) FEEDS exports the items. JSON Lines, CSV, JSON and XML all work, or you can pass -O books.csv on the command line instead.
(4) response.follow() resolves relative URLs the way urljoin() did in the Requests version.
Scrapy only fetches HTML, so for pages that need a browser, the scrapy-playwright plugin renders requests in Playwright from inside a spider. To crawl through ScrapingBee instead, use the scrapingbee crawl command in the ScrapingBee CLI (command-line tool), which is built on Scrapy.
It can start from a URL or a sitemap (an XML file that lists a site's pages) and follow same-site links, sending any API parameter with each request. In my test, scrapingbee crawl https://books.toscrape.com/ --render-js false --max-pages 3 saved three pages and a manifest (a JSON file that maps each URL to its saved page).
Our guide to crawling with Python takes a broader look at crawling.
2. Fetch many pages concurrently with asyncio and httpx
When you don't want a framework, asyncio lets one Python process wait on many network requests at the same time. Install httpx, the HTTP client this example uses, with uv add httpx.
The catch is that unlimited concurrency (many requests running at once) can overload a small site. The key line is the Semaphore, which caps the script at five requests in flight (sent but not yet answered).
The script fetches all 50 Books to Scrape listing pages with httpx's AsyncClient. Unlike the pagination loop, it builds the page URLs directly, which is safe here because that loop already showed the catalog has exactly 50 pages:
import asyncio
import time
import httpx
from bs4 import BeautifulSoup
PAGE_URL = "https://books.toscrape.com/catalogue/page-{}.html"
HEADERS = {"User-Agent": "my-first-scraper/1.0 (+https://example.com/contact)"}
MAX_CONCURRENCY = 5 # at most 5 requests in flight at once
def parse_titles(html: str) -> list[str]:
soup = BeautifulSoup(html, "lxml")
return [link["title"] for link in soup.select("article.product_pod h3 a")]
async def fetch_titles(client: httpx.AsyncClient, limit: asyncio.Semaphore, url: str) -> list[str]:
async with limit:
response = await client.get(url)
response.raise_for_status()
return parse_titles(response.text)
async def main() -> list[str]:
limit = asyncio.Semaphore(MAX_CONCURRENCY)
async with httpx.AsyncClient(headers=HEADERS, timeout=10) as client:
pages = await asyncio.gather(
*(fetch_titles(client, limit, PAGE_URL.format(n)) for n in range(1, 51))
)
return [title for page in pages for title in page]
start = time.perf_counter()
titles = asyncio.run(main())
print(f"{len(titles)} titles from 50 pages in {time.perf_counter() - start:.1f} s")
It finished in 3.5 seconds:
1000 titles from 50 pages in 3.5 s
For comparison, the same 50 pages took 19.4 seconds one by one with Requests and no delay, and 74.7 seconds in the polite pagination loop from earlier. Five requests in flight with httpx were about 5.5x faster than one at a time with Requests, but they also put five times the load on the site.
The load is fine here, because Books to Scrape exists to be scraped. On a small site you don't own, keep the cap at 5 or lower and add an await asyncio.sleep(1) inside the async with limit: block.
If you run this in a Jupyter notebook, replace asyncio.run(main()) with await main(), because the notebook already runs an event loop (the scheduler that drives async code). Async only helps while your scraper waits on the network. It doesn't speed up parsing, and the parsing here still runs one page at a time.
Picking an async HTTP client
Other libraries fit here too. aiohttp is async-only and actively maintained (3.14.5, October 2026). httpx also works without async and supports HTTP/2. Its last stable release, 0.28.1, came out in December 2024, and the 1.0 pre-releases that started in July 2025 haven't produced a stable version.
Pydantic (the team behind the popular data-validation library) now maintains httpx2, a fork of httpx. Version 2.16.0 came out in October 2026. The Anthropic SDK already depends on httpx2.
After changing the import to import httpx2 as httpx, the rest of the script ran unchanged in 3.3 seconds. More async patterns are in our guide to asynchronous scraping in Python.
Our roundup of Python HTTP clients compares the options in depth.
Why Python web scrapers get blocked or break
A scraper that worked last week can suddenly return a 403, a CAPTCHA page or an empty list. It's either blocked because the site decided the request came from a bot, or broken because the page changed under it.
The signals that give a Python scraper away
Websites block scrapers by checking the whole request: headers, the TLS fingerprint (the encryption settings your client offers when it connects), the HTTP version and the IP address.
Plain Requests sends a Python TLS fingerprint over HTTP/1.1, often from a datacenter IP, and many sites also run JavaScript challenges. A browser-like User-Agent alone rarely helps. Slow down first, then switch to curl_cffi, a real browser or a scraping API.
Any one of those signals can flag your scraper:
- The User-Agent header: Requests announces itself as python-requests/2.34.2. Some sites block that string outright. Our guide to headers for web scraping covers the rest of the header set.
- Missing or oddly ordered headers: Browsers send a dozen headers such as Accept-Language and the Sec-Fetch-* family, in a consistent order. A request with three headers, or with Chrome's User-Agent and no Chrome headers, stands out.
- The TLS fingerprint (JA3 or JA4): Before any HTTP is sent, the client and server agree on encryption. The list of ciphers (encryption methods) and extensions your client offers is distinctive enough to hash into a short ID. JA3 and JA4 are two common formats for that ID. Python's ssl module produces a hash no browser has, and no header can change it.
- The HTTP/2 fingerprint: Browsers speak HTTP/2 with characteristic settings and frame order. Requests only speaks HTTP/1.1, which on its own is unusual for a "Chrome" in 2026.
- IP reputation: Requests from cloud and datacenter ranges get far more scrutiny than home connections. A single IP sending thousands of requests gets rate-limited (slowed down or cut off) whatever it looks like.
- JavaScript challenges and behavior checks: Services such as Cloudflare, DataDome, Akamai, HUMAN and Kasada serve a page that runs JavaScript to check for a real browser. Some also watch mouse movement and timing. An HTTP client can't run these checks.
Sites have good reason to check for bots. Imperva's 2026 Bad Bot Report says automated traffic made up more than 53% of all web traffic in 2025, by its own measurement. Cloudflare has blocked AI crawlers by default on new domains since July 2025.
From September 15, 2026, its default for new domains blocks training and agent crawlers on pages that show ads (details). Bot defenses can class a tutorial script as a bot no matter what it's for.
Polite defaults that keep a scraper welcome
Many blocks are self-inflicted: too many requests, too fast, with retries that pile more load onto a server that's already struggling.
These defaults help you avoid them:
- Use one Session with an honest User-Agent and a timeout on every request.
- Wait between requests. One second is a sensible floor. Honor Crawl-delay when robots.txt sets one.
- Retry only 429 and 5xx responses, with exponential backoff (waits that double each time). These errors are temporary. A 403 isn't, so retrying it only makes you look more like a bot.
- Honor the Retry-After header.
- Stop after repeated 403 or 429 responses instead of pushing through them.
This crawler applies all five, using the Retry class from urllib3, the networking library Requests is built on:
import time
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
HEADERS = {"User-Agent": "my-first-scraper/1.0 (+https://example.com/contact)"}
def polite_session() -> requests.Session:
retry = Retry(
total=3,
backoff_factor=1, # retries at once, then after 2 s and 4 s
status_forcelist=[429, 500, 502, 503, 504],
allowed_methods=["GET"],
respect_retry_after_header=True, # a server's Retry-After wins over backoff
raise_on_status=False, # hand back the last response instead of raising
)
session = requests.Session()
session.headers.update(HEADERS)
session.mount("https://", HTTPAdapter(max_retries=retry))
session.mount("http://", HTTPAdapter(max_retries=retry))
return session
def crawl(urls: list[str], delay: float = 1.0, max_blocked: int = 3):
blocked = 0
with polite_session() as session:
for url in urls:
started = time.perf_counter()
response = session.get(url, timeout=(5, 15))
print(f"{response.status_code} {url} ({time.perf_counter() - started:.1f} s)")
if response.status_code in (403, 429):
blocked += 1
if blocked >= max_blocked:
print(f"Stopping: {blocked} blocked responses in a row. Slow down or rethink the approach.")
return
else:
blocked = 0
if response.ok: # a 5xx that survived the retries is skipped, not kept
yield response
time.sleep(delay)
pages = list(crawl([
"https://httpbin.org/status/200",
"https://httpbin.org/status/503", # retried 3 times with backoff, then given up
"https://httpbin.org/status/403", # not retried: a 403 won't fix itself
"https://httpbin.org/status/403",
"https://httpbin.org/status/403",
"https://httpbin.org/status/200", # never reached
]))
print(f"Kept {len(pages)} page(s)")
The test URLs on httpbin.org show each rule firing:
200 https://httpbin.org/status/200 (0.9 s)
503 https://httpbin.org/status/503 (6.9 s)
403 https://httpbin.org/status/403 (0.2 s)
403 https://httpbin.org/status/403 (0.2 s)
403 https://httpbin.org/status/403 (0.2 s)
Stopping: 3 blocked responses in a row. Slow down or rethink the approach.
Kept 1 page(s)
The 503 took 6.9 seconds because urllib3 retried it three times. Its backoff skips the first wait, so with backoff_factor=1 the first retry came immediately, and the next two came after 2 and 4 seconds.
You might expect waits of 1, 2 and 4 seconds instead. I also checked respect_retry_after_header against a local server that answered 429 with Retry-After: 3. urllib3 waited exactly 3 seconds before each retry instead of using its own backoff.
Debugging a scraper that stops working
When a scraper breaks, find out what happened before you change any code. The cause is one of four things: the request failed, the site blocked it, the page needs JavaScript, or the layout changed.
The status code is the first clue:
| What you get | What it usually means | What to do |
|---|---|---|
| 403 Forbidden | Blocked, or the path isn't allowed | Slow down and check your headers. If it persists on a public page, try curl_cffi or a scraping API |
| 404 Not Found | Wrong URL, or the page was removed | Check the URL the scraper built, especially relative links |
| 429 Too Many Requests | You're going too fast | Read Retry-After, wait, and lower your rate for good |
| 5xx Server Error | The server is struggling | Retry later with backoff; don't add load |
| 200 with a challenge page | A bot check answered instead of the site | Look for "Just a moment" or a CAPTCHA in the saved HTML; a plain HTTP client can't pass it |
| 200 with no data | JavaScript renders it, or the layout changed | Compare the saved HTML with what you see in the browser |
A common breakage is the last row of that table: the site renamed a class, and your selector now matches nothing. A scraper that writes an empty CSV without complaint is worse than one that crashes, so make yours fail loudly and save the evidence:
from pathlib import Path
import requests
from bs4 import BeautifulSoup
HEADERS = {"User-Agent": "my-first-scraper/1.0 (+https://example.com/contact)"}
CHALLENGE_MARKERS = ("just a moment", "captcha", "access denied", "verify you are human")
def select_or_fail(html: str, selector: str, url: str, minimum: int = 1) -> list:
found = BeautifulSoup(html, "lxml").select(selector)
if len(found) >= minimum:
return found
Path("debug_page.html").write_text(html, encoding="utf-8")
if any(marker in html.lower() for marker in CHALLENGE_MARKERS):
raise RuntimeError(f"{url} returned a bot challenge, not the page. Saved debug_page.html")
raise RuntimeError(f"{selector!r} matched {len(found)} elements on {url} (expected {minimum}+). "
"The layout changed or the data needs JavaScript. Saved debug_page.html")
url = "https://quotes.toscrape.com/js/"
response = requests.get(url, headers=HEADERS, timeout=10)
quotes = select_or_fail(response.text, "div.quote", url, minimum=10)
When I pointed the script at the JavaScript quotes page with a plain request, it raised a clear error and left the HTML behind to inspect:
RuntimeError: 'div.quote' matched 0 elements on https://quotes.toscrape.com/js/ (expected 10+). The layout changed or the data needs JavaScript. Saved debug_page.html
Three more habits make the next breakage quicker to fix. First, save a copy of a page (a fixture) while the scraper still works, with two lines at the end of main() in hn_scraper.py:
Path("fixtures").mkdir(exist_ok=True)
Path("fixtures/hn_front_page.html").write_text(html, encoding="utf-8")
Add from pathlib import Path to the imports in hn_scraper.py. Second, write a test that runs your parser against that saved page, with no network involved, every time you change the code.
Testing the parser against the saved page
Install pytest with uv add --dev pytest, save this as test_parser.py and run uv run pytest:
from pathlib import Path
from hn_scraper import parse_stories
def test_parse_stories_reads_a_saved_front_page():
html = Path("fixtures/hn_front_page.html").read_text(encoding="utf-8")
stories = parse_stories(html, limit=30)
assert len(stories) == 30
assert all(story.title and story.url.startswith("http") for story in stories)
assert sum(story.points for story in stories) > 0
On the saved front page, all 30 stories parsed:
1 passed in 0.12s
Third, when a browser-based scraper misbehaves, watch it. Launching Playwright with headless=False opens a visible window, and slow_mo=500 pauses half a second between actions so you can see where it goes wrong:
browser = p.chromium.launch(headless=False, slow_mo=500)
When a scraping API renders the page for you, ask it for a screenshot of what its browser saw.
What to try next, in order
When a public page keeps blocking you, escalate one step at a time, because each step costs more than the last:
- Slow down and identify yourself: Add the polite defaults.
- Switch to curl_cffi: It fixes the TLS and HTTP/2 fingerprint with a one-line change and keeps HTTP-client speed.
- Use a real browser: Playwright runs JavaScript challenges that HTTP clients can't. For sites that detect automation, stealth browsers such as nodriver hide the signs a standard automated browser leaves behind. Camoufox is another option.
- Hand it to a web scraping API: Instead of running browsers and rotating proxies yourself, a service like ScrapingBee does both. It avoids CAPTCHAs by rendering pages with clean browsers and proxies. It doesn't solve them.
Our guide to scraping without getting blocked has the full playbook.
How to call a web scraping API from Python
A web scraping API fetches the page for you, renders JavaScript, rotates proxies, then returns HTML, Markdown, or extracted JSON to your Python code. You trade a per-request cost for not running browsers or proxies yourself.
Use one when pages block you or need JavaScript, or when you need many pages reliably. We'll use ScrapingBee here, since I tested every API example against it.
1. Fetch a page with the ScrapingBee SDK or Requests
Start by creating a free account to get an API key with 1,000 credits.
Then, put the key in an environment variable (a named setting your shell passes to programs) so it never ends up in your code or your repository:
export SCRAPINGBEE_API_KEY="your-api-key" # macOS and Linux
On Windows PowerShell, the first line sets the key for the current window.
setx saves it for terminals you open later, but not for the current one:
$env:SCRAPINGBEE_API_KEY = "your-api-key" # this terminal
setx SCRAPINGBEE_API_KEY "your-api-key" # new terminals from now on
The quickest start is the official Python SDK, installed with uv add scrapingbee (or pip install scrapingbee). Use html_api().
In SDK 2.2.0, the older client.get() still works, but it's deprecated, and version 3.0.0 will remove it:
import os
from scrapingbee import ScrapingBeeClient
client = ScrapingBeeClient(api_key=os.environ["SCRAPINGBEE_API_KEY"])
response = client.html_api(
"https://quotes.toscrape.com/js/",
params={"wait_for": "div.quote"}, # JavaScript rendering is on by default
timeout=150,
)
print(response.status_code, "| credits:", response.headers.get("Spb-cost"))
print(response.text.count('class="quote"'), "quotes in the rendered HTML")
The quotes that Requests couldn't see are in the rendered HTML:
200 | credits: 5
10 quotes in the rendered HTML
You don't need the SDK. The API is one HTTP endpoint, so plain Requests works the same way, with the key in an Authorization: Bearer header. Older examples put api_key in the URL, but that parameter is deprecated.
A key in a query string also ends up in server logs and proxy histories:
import os
import requests
API_URL = "https://app.scrapingbee.com/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['SCRAPINGBEE_API_KEY']}"}
response = requests.get(
API_URL,
headers=HEADERS,
params={"url": "https://books.toscrape.com/", "render_js": "false"}, # static page: 1 credit
timeout=150,
)
response.raise_for_status()
print(response.status_code, "| credits:", response.headers.get("Spb-cost"))
print(response.text.count('class="product_pod"'), "books in the HTML")
Note the timeout=150: ScrapingBee keeps working on a request for up to 140 seconds by default, so your client should wait a little longer. The request printed:
200 | credits: 1
20 books in the HTML
2. Turn off JavaScript rendering for static pages
JavaScript rendering is on by default, but a static page doesn't need it. That's why the Requests example in step 1 sets render_js=false: Books to Scrape has all its data in the HTML, so the request cost 1 credit instead of 5.
These are the per-request costs from ScrapingBee's documentation. The 1-credit, 5-credit and AI costs matched the Spb-cost header on my own requests:
| Setup | Credits per request |
|---|---|
| Classic proxy, no JavaScript (render_js=false) | 1 |
| Classic proxy with JavaScript (the default) | 5 |
| Premium (residential) proxy, no JavaScript | 10 |
| Premium proxy with JavaScript | 25 |
| Stealth proxy (JavaScript only) | 75 |
| AI extraction (ai_query, ai_extract_rules) | +5 on top |
Billing depends on the status code ScrapingBee returns, not the target site's. You aren't charged for 500. ScrapingBee's own 400 (bad parameters), 401 (no credits or a bad key), 403 (a banned account or blocked destination) and 429 (too many concurrent requests) are also free.
By default, a target's own 403 or 503 reaches you as a 500, so a page that blocks the request doesn't cost anything. Add transparent_status_code=true if you need the site's real status. ScrapingBee does charge for a 200, 404, 410 or 413.
3. Let Auto Mode find the best setup that works
When you don't know what a page needs, mode=auto tries cheaper configurations first and moves up only if they fail. Add max_cost=25, and it stops before the 75-credit Stealth tier. If every allowed setup fails, the request costs nothing.
Two response headers tell you what happened. Spb-cost is what the request cost, and Spb-auto-cost is the cost of the configuration that succeeded, or 0 if none did:
import os
import requests
API_URL = "https://app.scrapingbee.com/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['SCRAPINGBEE_API_KEY']}"}
for url, wait_for in [("https://books.toscrape.com/", None),
("https://quotes.toscrape.com/js/", "div.quote")]:
params = {"url": url, "mode": "auto", "max_cost": "25"}
if wait_for:
params["wait_for"] = wait_for # Auto Mode won't wait for late content on its own
response = requests.get(API_URL, headers=HEADERS, params=params, timeout=150)
print(url)
print(" status:", response.status_code,
"| Spb-cost:", response.headers.get("Spb-cost"),
"| Spb-auto-cost:", response.headers.get("Spb-auto-cost"))
Auto Mode picked the 1-credit setup for the static page and the 5-credit rendered setup for the JavaScript page:
https://books.toscrape.com/
status: 200 | Spb-cost: 1 | Spb-auto-cost: 1
https://quotes.toscrape.com/js/
status: 200 | Spb-cost: 5 | Spb-auto-cost: 5
One thing I learned the hard way: Auto Mode decides how to fetch the page, not when it finishes. On quotes.toscrape.com/js-delayed/, which renders after a delay, Auto Mode without wait_for returned a 200 with zero quotes and still charged 5 credits.
With wait_for="div.quote", it returned all 10. If a page loads its data late, pass wait_for yourself. You can't combine Auto Mode with render_js, premium_proxy, stealth_proxy or transparent_status_code. Sending any of them returns a 400.
Auto Mode also doesn't always pick the cheapest setup that would work. On the static Quotes to Scrape homepage, Auto Mode chose the 5-credit rendered setup on every request in the fetch test, with or without wait_for.
With render_js=false, the page returned the same 10 quotes for 1 credit. When you already know a page is static, set render_js=false yourself.
4. Get JSON back with extract rules or a plain-English query
If you already know the CSS selectors, extract_rules returns JSON instead of HTML, so there's no parsing on your side:
import json
import os
import requests
API_URL = "https://app.scrapingbee.com/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['SCRAPINGBEE_API_KEY']}"}
extract_rules = {
"quotes": {
"selector": "div.quote",
"type": "list",
"output": {"text": ".text", "author": ".author", "tags": {"selector": ".tag", "type": "list"}},
}
}
response = requests.get(
API_URL,
headers=HEADERS,
params={
"url": "https://quotes.toscrape.com/js/",
"wait_for": "div.quote",
"extract_rules": json.dumps(extract_rules),
},
timeout=150,
)
response.raise_for_status()
data = response.json()
print(len(data["quotes"]), "quotes | credits:", response.headers.get("Spb-cost"))
print(json.dumps(data["quotes"][0], indent=2, ensure_ascii=False))
The API rendered the page, applied the selectors and returned structured data:
10 quotes | credits: 5
{
"text": "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”",
"author": "Albert Einstein",
"tags": [
"change",
"deep-thoughts",
"thinking",
"world"
]
}
When writing selectors isn't worth it, describe the fields in plain English with ai_extract_rules (or ask a single question with ai_query). Each adds 5 credits to the request. You can give a field a description and a type:
import json
import os
import requests
API_URL = "https://app.scrapingbee.com/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['SCRAPINGBEE_API_KEY']}"}
ai_rules = {
"title": "the book title",
"price": {"description": "price including tax, as a number", "type": "number"},
"stock": {"description": "how many copies are in stock", "type": "number"},
}
response = requests.get(
API_URL,
headers=HEADERS,
params={
"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
"render_js": "false",
"ai_extract_rules": json.dumps(ai_rules),
},
timeout=150,
)
response.raise_for_status()
print(response.json(), "| credits:", response.headers.get("Spb-cost"))
On a Books to Scrape product page, it returned:
{'title': 'A Light in the Attic', 'price': 51.77, 'stock': 22} | credits: 6
Use CSS rules for stable pages you scrape often, since they're cheaper and predictable. Switch to AI rules for messy or changing pages where maintaining selectors costs more than 5 credits a page. Either way, spot-check the output.
Our page on AI data extraction covers the AI options in more detail.
5. Return Markdown or a screenshot instead of HTML
return_page_markdown=true returns clean Markdown, the format to feed a large language model (LLM). It also suits a retrieval-augmented generation (RAG) index, which an LLM searches before it answers.
screenshot=true returns a PNG of what the renderer saw, which is the fastest way to debug a selector that comes back empty:
import os
from pathlib import Path
import requests
API_URL = "https://app.scrapingbee.com/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['SCRAPINGBEE_API_KEY']}"}
TARGET = "https://quotes.toscrape.com/js/"
# Markdown: clean text for an LLM or a RAG index
markdown = requests.get(
API_URL, headers=HEADERS, timeout=150,
params={"url": TARGET, "wait_for": "div.quote", "return_page_markdown": "true"},
)
markdown.raise_for_status()
print(markdown.text[:180])
# Screenshot: see what the renderer saw when a selector comes back empty
shot = requests.get(
API_URL, headers=HEADERS, timeout=150,
params={"url": TARGET, "wait_for": "div.quote", "screenshot": "true"},
)
shot.raise_for_status()
Path("debug_screenshot.png").write_bytes(shot.content)
print("Saved debug_screenshot.png,", len(shot.content) // 1024, "KB")
The script printed the start of the Markdown and saved the PNG:
[Quotes to Scrape](/)
=====================
[Login](/login)
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by
Saved debug_screenshot.png, 122 KB

ScrapingBee also has dedicated endpoints that return parsed JSON for specific sources, such as YouTube search and video metadata or Walmart products, so you don't maintain selectors for them.
Whatever you scrape, keep to public pages. ScrapingBee's Acceptable Use Policy doesn't allow scraping behind a login. The Python SDK guide and the full API documentation cover every parameter. For Markdown specifically, see our Markdown output page.
How 6 Python web scraping setups and 6 HTML parsers performed in tests across 8 sites
Every recommendation in this guide makes a claim about speed, cost or reliability, and published benchmarks often disagree. So I measured them myself, with the same pages, the same pacing and the same success test for every tool.
I ran two benchmarks. The first fetched the same 8 public pages with 6 setups and checked whether the data came back. The second timed 6 HTML parsers on three saved files. Everything ran on November 1, 2026.
How I ran the tests
The fetch test compared six setups from five tools:
| Setting | What I used |
|---|---|
| Clients | Requests 2.34.2, httpx 0.28.1, curl_cffi 0.17.0 (impersonate="chrome"), Playwright 1.64.1 with headless Chromium, and ScrapingBee in two setups: render_js=false, and mode=auto with max_cost=25 and wait_for |
| Sites | Books to Scrape, quotes.toscrape.com with its /js/ and /js-delayed/ versions, the scrapethissite.com forms page, the webscraper.io test shop, the Hacker News front page and a Wikipedia table page. Each site's robots.txt allowed every path I requested |
| Volume | 50 requests per client per site in each of 2 rounds on November 1, 2026 (starting 17:56 and 21:00 UTC). Hacker News and Wikipedia, which aren't practice sites, got 10 per round |
| Pacing | One request at a time per site, at least a second apart, and 30 seconds apart on Hacker News, as its robots.txt asks. Clients took turns request by request |
| User-Agent | Requests and httpx sent a contact URL. curl_cffi and Playwright kept their browser User-Agent. On Wikipedia, every client sent the contact User-Agent its policy asks for |
| Success | The expected data in the result: 20 book titles, 10 quotes, 25 hockey teams, 3 products, 30 stories or at least 200 table rows. A 200 status alone didn't count |
| Browser timing | Chromium launched once per run, with a fresh, empty browser context (like a new incognito window) for every request, so no page came from cache |
| Metrics | Success rate per site, median and 95th-percentile time per successful page, peak memory and lines of code of a one-page scraper, and ScrapingBee credits from the Spb-cost header |
| Machine | MacBook Pro with an Apple M3 Max, macOS 26.7.1, Python 3.15.2 |
| Network | The Python clients went through a local security proxy (Safe-chain, which checks package installs), and Playwright's browser didn't. The proxy doesn't change the TLS fingerprint or the time of a reused connection, and adds 0.2 s to a new one. A DNS outage from 21:17 to 21:21 UTC failed 7 requests, which I counted as failures |
Which tools got the data on each site
Apart from that outage, every setup got the data on every static page, and only the setups that render pages in a browser got it on the JavaScript pages:

| Setup | Books to Scrape | Quotes to Scrape | Scrapethissite forms | webscraper.io shop | Hacker News | Wikipedia table | Quotes /js/ | Quotes /js-delayed/ |
|---|---|---|---|---|---|---|---|---|
| Requests | 100% | 100% | 100% | 100% | 95% | 100% | 0% | 0% |
| httpx | 100% | 100% | 100% | 100% | 95% | 100% | 0% | 0% |
| curl_cffi (chrome) | 100% | 100% | 100% | 100% | 100% | 100% | 0% | 0% |
| Playwright (Chromium) | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
| ScrapingBee, no JS | 100% | 100% | 100% | 100% | 95% | 100% | 0% | 0% |
| ScrapingBee, Auto Mode | 100% | 100% | 100% | 100% | 95% | 100% | 100% | 99% |
Three findings stand out:
(1) Plain Requests was enough for every static page: Requests and httpx each got the data on 439 of 440 requests to the six static pages, and curl_cffi on all 440. Requests and httpx each missed once, during the outage.
None of these sites run bot protection against an honest scraper, so the test shows where each tool stops working, not how it fares against anti-bot systems.
(2) The JavaScript pages split the field in two. Requests, httpx, curl_cffi and ScrapingBee without rendering never got a quote. Every response they received was a 200 with an empty quote list.
Playwright returned all 10 quotes on all 200 of its requests. Auto Mode did the same on 199, and the one miss timed out during the outage.
(3) The browser is an expensive option. Playwright was the slowest setup on static pages, and its one-page scraper needed about 10x the memory of Requests. httpx looks twice as slow as Requests only because it closes idle connections after 5 seconds by default, and each client here waits 6 to 8 seconds between turns. With keepalive_expiry=30, httpx matched Requests. Then format keepalive_expiry=30 as code.
The speed and cost numbers behind those bullets:

| Setup | Median, static pages | p95, static pages | Median, /js/ | Median, /js-delayed/ | Peak memory, one page | Lines of code | Credits used |
|---|---|---|---|---|---|---|---|
| Requests | 0.40 s | 1.03 s | – | – | 48.5 MB | 7 | – |
| httpx | 0.92 s | 1.58 s | – | – | 54.9 MB | 7 | – |
| curl_cffi (chrome) | 0.35 s | 1.06 s | – | – | 52.3 MB | 7 | – |
| Playwright (Chromium) | 1.63 s | 2.26 s | 1.93 s | 12.3 s | 502.5 MB | 10 | – |
| ScrapingBee, no JS | 0.99 s | 2.28 s | – | – | 48.4 MB | 13 | 638 |
| ScrapingBee, Auto Mode | 1.14 s | 2.97 s | 2.17 s | 12.1 s | 48.4 MB | 13 | 1834 |
Raw rows, one per request, are in the repo's results folder.
Parser speed: html.parser vs lxml vs selectolax
The parser test timed six parsing methods on the same three files and extracted the same fields. The files were a Books to Scrape listing page (50 KB), a long Wikipedia article (2 MB) and a generated 1 MB product grid.
Each combination ran 1,000 times with timeit (Python's built-in timing module), and I checked that all six returned identical data before timing anything. The versions were beautifulsoup4 4.15.0, lxml 6.1.3, html5lib 1.1, parsel 1.12.1 and selectolax 1.0.2:

| Parser | Books listing (50 KB) | Wikipedia article (2 MB) | Generated grid (1 MB) | Peak memory added (2 MB file) |
|---|---|---|---|---|
| Beautiful Soup + html.parser | 6.91 ms | 172.3 ms | 216.5 ms | 21.7 MB |
| Beautiful Soup + html5lib | 12.59 ms | 388.8 ms | 397.2 ms | 29.5 MB |
| Beautiful Soup + lxml | 5.76 ms | 129.9 ms | 172.4 ms | 22.0 MB |
| parsel | 0.59 ms | 29.4 ms | 53.7 ms | 21.9 MB |
| lxml.html | 0.61 ms | 27.3 ms | 23.8 ms | 18.6 MB |
| selectolax (lexbor) | 0.17 ms | 8.1 ms | 6.4 ms | 21.5 MB |
The surprise for me was how little the parser choice inside Beautiful Soup matters. Switching Beautiful Soup from html.parser to lxml saved 17% to 25%, because most of the time goes into Beautiful Soup building its own Python objects for every tag.
Compared with Beautiful Soup on html.parser, lxml.html on its own was 6 to 11 times faster. selectolax was 21 to 42 times faster. html5lib, the parser most tolerant of broken HTML, was about twice as slow as html.parser and needed about a third more memory on the Wikipedia page. The other five were within a few megabytes of each other.
For a few hundred pages, the parser choice changes nothing: 300 Wikipedia-sized pages cost about 52 seconds of parsing with Beautiful Soup, and the network takes longer than that. The parser starts to matter at hundreds of thousands of pages.
At a million pages of that size, Beautiful Soup with html.parser would take about 48 hours to parse, and selectolax about 2.3 hours. Start with Beautiful Soup for its easy API. When a profiler (a tool that measures where your code spends time) shows parsing is the slow part, switch to selectolax or lxml.html.
My selectolax-to-lxml.html ratio (3.4 to 3.7 times faster) is close to the benchmark in the Art of Web Scraping book, which timed parsing alone and found about 4 times. Our guide to making Beautiful Soup faster covers the tricks that help when you'd rather keep Beautiful Soup.
Python web scraping with LLMs and AI coding assistants
ChatGPT, Claude and other assistants can write and fix scraper code. With browsing or tools, they can also read a few pages, but they aren't built to pull structured data from thousands of pages on a schedule.
The pattern that works is Python fetching the pages and an LLM writing selectors or extracting fields, with you checking the output.
What ChatGPT and coding assistants can do
Assistants handle the tedious parts well. They can turn a pasted HTML snippet into selectors, explain a traceback (Python's error report) or convert a cURL command into Requests code. They can also write the dataclass for your fields. The risk is code that works on one page and fails on the next, because the assistant only sees the page you gave it.
I tested how much that matters. I gave a coding assistant (Claude Sonnet 5.5) the saved HTML of one page: the laptops category of the webscraper.io test shop. The prompt asked for a function that returns every product's name, price, description and review count.
Then I ran its code on three pages it had never seen: tablets, touch phones and the home page. It got the name and price of all 150 products right on all four pages, with no empty fields. The assistant also handled a detail I'd have expected it to miss.
On 98 of the 117 laptops, the visible name is cut off ("Asus VivoBook..."), and its code reads the full name from the link's title attribute instead.
That result is the best case, though: every page on that site uses the same template. Other sites have sale prices, missing review counts and one-off layouts that the sample page didn't show. So treat AI-written selectors like any other code and run them against several pages, including an odd one, before you trust them.
Extracting fields with an LLM instead of selectors
For pages that change layout often, or where the fields are buried in prose, you can skip selectors and have a model read the page. Send Markdown or cleaned HTML rather than raw HTML. Both use a fraction of the tokens (the text units models read and bill by) and are easier for the model to read.
This script fetches a product page as Markdown through ScrapingBee and asks Claude to fill a Pydantic model (a Python class that defines your fields and their types). The call uses the Anthropic SDK's structured outputs, which validate the reply against that class.
The script needs uv add anthropic and an ANTHROPIC_API_KEY environment variable:
import os
import anthropic
import requests
from pydantic import BaseModel
class Book(BaseModel):
title: str
price_gbp: float
in_stock: int
upc: str
# 1. Fetch the page as Markdown (static page, so no JavaScript rendering: 1 credit)
page = requests.get(
"https://app.scrapingbee.com/api/v1",
headers={"Authorization": f"Bearer {os.environ['SCRAPINGBEE_API_KEY']}"},
params={
"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
"render_js": "false",
"return_page_markdown": "true",
},
timeout=150,
)
page.raise_for_status()
# 2. Ask Claude to fill the schema; parse() validates the reply against it
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY
response = client.messages.parse(
model="claude-opus-5-5",
max_tokens=1024,
output_config={"effort": "low"},
messages=[{"role": "user", "content": f"Extract the book details from this page:\n\n{page.text}"}],
output_format=Book,
)
book = response.parsed_output
print(book)
print(f"{len(page.text):,} characters of Markdown, "
f"{response.usage.input_tokens:,} input tokens, {response.usage.output_tokens:,} output tokens")
It returned a validated Book object:
title='A Light in the Attic' price_gbp=51.77 in_stock=22 upc='a897fe39b1053632'
1,770 characters of Markdown, 1,014 input tokens, 50 output tokens
Two rules keep LLM extraction honest. Validate against a schema, so a missing or malformed field raises an error instead of slipping into your data. And price it before you scale it. Each page costs model tokens on top of the fetch, so for a stable layout, CSS selectors or extract_rules are cheaper and return the same result every time.
If you'd rather not run a model yourself, ScrapingBee's ai_query and ai_extract_rules handle the extraction for 5 extra credits per request.
Storing scraped pages for RAG and AI agents
If scraped pages will feed a RAG index or an AI agent (an LLM that uses tools), store them so every answer can be traced back to its source:
- Markdown for page content: Markdown keeps clean text with the headings, so it splits well into chunks (the short passages a RAG index stores).
- JSON Lines for the collection: It stores one document per line so that you can append new pages.
- Metadata on every record: Store at least the source URL and when you scraped it, so you can cite the page and know when to refresh it.
A record in that collection looks like this:
{"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "scraped_at": "2026-11-01T18:20:00+00:00", "title": "A Light in the Attic", "content": "# A Light in the Attic\n\n£51.77\n\nIn stock (22 available)..."}
Keep the raw page too if you can afford the storage, so a later chunking or extraction bug doesn't mean scraping again.
Connecting AI assistants through MCP and the CLI
The Model Context Protocol (MCP) lets AI assistants call external tools. ScrapingBee runs an MCP server, so assistants such as Claude or Cursor can fetch pages through your account.
In Claude Code, connecting takes one command. Your shell fills in the key from the environment variable you set earlier, and Claude Code saves it in its own MCP config:
claude mcp add --transport http scrapingbee https://mcp.scrapingbee.com/mcp \
--header "Authorization: Bearer $SCRAPINGBEE_API_KEY"
When I connected to it on November 1, 2026, the server (version 2.2.1) offered 18 tools, including get_page_html, get_page_text, extract_page_data and get_screenshot.
For coding agents that work in a terminal, the ScrapingBee CLI fetches pages directly from the shell. The docs recommend installing it with uv, and it picks up the SCRAPINGBEE_API_KEY you exported earlier:
uv tool install scrapingbee-cli
scrapingbee scrape "https://books.toscrape.com/" --render-js false --return-page-markdown true --output-file books.md
It saved 10,455 bytes of Markdown to books.md, starting with the category menu:
[Home](index.html)
All products
* [Books](catalogue/category/books_1/index.html)
+ [Travel](catalogue/category/books/travel_2/index.html)
+ [Mystery](catalogue/category/books/mystery_3/index.html)
If your assistant writes spiders, Scrapy now offers an official agent plugin and a Scrapy MCP server, both covered on its coding agents page. Whichever route you take, keep your API key in an environment variable or the tool's own config. Never paste it into a chat with an assistant, where it can end up in logs and chat history.
For an open-source alternative built around LLM-ready output, see our Crawl4AI guide.
Scale your web scraping infrastructure with ScrapingBee
From the start, your goal was the data. Running a fleet of headless browsers and tracking which TLS fingerprint got flagged is overhead you didn't sign up for. Requests and Beautiful Soup will carry you a long way, and for static pages they're all you need. When pages start blocking you or the volume climbs, the infrastructure becomes the project.
ScrapingBee takes that infrastructure off your hands:
- JavaScript rendering on demand: Headless browsers on ScrapingBee's servers render the page, and wait_for waits for late content.
- Auto Mode with a cost cap: Auto Mode tries more expensive setups only as far as max_cost allows and charges nothing when every attempt fails.
- Proxies you don't manage: Classic, premium residential and stealth pools rotate for you, with country targeting.
- Structured output: Get JSON from CSS extract rules or plain-English AI extraction, Markdown for LLMs, or screenshots for debugging.
- Works where you work: Call it from the Python SDK, plain Requests or the CLI, or connect AI assistants through the MCP server.
Start with 1,000 free API credits with no credit card required and render your first JavaScript page without installing a browser.
Frequently asked questions about Python web scraping
What is the best Python library for web scraping?
There isn't one best library, because each solves a different problem. Requests with Beautiful Soup is the easiest start for static HTML. Playwright handles pages that render with JavaScript, and Scrapy is built for crawling thousands of pages. curl_cffi helps when a public site rejects Python's TLS fingerprint. Pick the library that fits the page you're scraping.
Is web scraping illegal?
Scraping publicly available data is generally lawful in many countries, but the answer depends on the data, the method and the jurisdiction. Scraping behind a login, circumventing bot protection, ignoring a site's terms or collecting personal data all add legal risk. For anything sensitive or commercial, get legal advice for your situation before you start.
Why does my Python scraper get a 403 error?
A 403 usually means the site recognized the request as automated or doesn't allow that path. Common causes are the default python-requests User-Agent, a Python TLS fingerprint, a datacenter IP address, or too many requests too quickly. Slow down and send honest headers first. If a public page still refuses, try curl_cffi or a scraping API.

