CLI

Installation

Recommended — install with uv (no virtual environment needed):

curl -LsSf https://astral.sh/uv/install.sh | sh
uv tool install scrapingbee-cli

Alternative — install with pip in a virtual environment:

pip install scrapingbee-cli

Verify the installation:

scrapingbee --version

Authentication

Save your API key so all commands can use it automatically.

Interactive prompt (recommended for first-time setup):

scrapingbee auth

Non-interactive (CI/CD, scripts):

scrapingbee auth --api-key YOUR_API_KEY

Environment variable (alternative — no file stored):

export SCRAPINGBEE_API_KEY=YOUR_API_KEY

The CLI also reads .env files in the current directory.

Show stored key location:

scrapingbee auth --show

Remove stored key:

scrapingbee logout

Shell Quoting

Always wrap JSON data and path expressions in single quotes ('...'). Double quotes trigger shell expansion that breaks your commands:

• $ triggers variable expansion — $HOME becomes /Users/sahil
• \ triggers escape sequences — \" becomes "
• ` triggers command substitution

This applies to: --ai-extract-rules, --extract-rules, --js-scenario, --smart-extract, --ai-query

# ✓ Correct — single quotes preserve JSON and special characters as-is
scrapingbee scrape "https://example.com" --ai-extract-rules '{"title":"product name","price":"price"}'

scrapingbee scrape "https://example.com" --js-scenario '{"instructions":[{"evaluate":"console.log(\"Hello World\");"}]}'

scrapingbee scrape "https://example.com" --smart-extract '...a[href=*mailto*].text'

# ✗ Wrong — double quotes cause shell expansion and break the JSON
scrapingbee scrape "https://example.com" --ai-extract-rules "{\"title\":\"product name\"}"

Tip: URLs should use double quotes (they may contain & or ?). JSON data and path expressions should use single quotes. When both appear, they sit side by side naturally:

scrapingbee scrape "https://example.com?page=1" --smart-extract '...h1'

Credits and Plan

Check your API credit balance and plan concurrency:

scrapingbee usage

For scripts, write the JSON response to a file:

scrapingbee usage --output-file usage.json

Example response:

{
  "max_api_credit": 1000000,
  "used_api_credit": 42150,
  "max_concurrency": 100,
  "current_concurrency": 0,
  "renewal_subscription_date": "2025-07-26T04:57:13.580067"
}

Interactive Mode

Run scrapingbee with no subcommand to enter an interactive REPL — a full-screen terminal UI with a pinned banner, live credit toolbar, virtual scrollback, and tab-completed commands.

scrapingbee

Think of it as a ScrapingBee workbench inside your terminal. Instead of retyping long commands, checking your dashboard for credits, opening output files manually, and losing context between runs, the REPL keeps the whole scraping session in one place:

• Experiment faster: run scrape, crawl, google, Amazon/Walmart/YouTube commands, and LLM scrapers without leaving the interface.
• Get command help as you type: completions cover commands, meta-commands, flags, boolean values, and choice values. The faint inline suggestion after your cursor is a history-aware preview of a likely continuation; press Right to accept the next word or End to accept all of it.
• See account state while you work: the responsive toolbar shows available credits, credits used in the current session, concurrency, refresh timing, active session defaults, and running-command elapsed time. On narrow terminals it automatically collapses to the highest-signal fields instead of wrapping.
• Reuse settings: set defaults once with :set country-code=fr or :set premium-proxy=true, then override them only when needed.
• Open and copy like a desktop app: existing local file/folder paths in output are underlined in yellow; click one to open it in Finder, Explorer, or your Linux desktop's default app. Drag across scrollback or :view to copy text to the clipboard.
• Inspect results in place: command output stays in scrollback, :view opens the last result in a built-in pager with pretty JSON/HTML detection, wrapped long lines, mouse-wheel scrolling, and an r raw/pretty toggle.
• Handle real workflows: paste multi-line commands into an editable buffer, check active schedules with :list, run shell commands with !command after enabling advanced features, and open the latest crawl log with :view crawl.
• Watch long jobs live: crawls and batches get a fixed status area with progress, fetched/saved counts, and the current crawl URL where available.

If you are exploring a site, tuning proxies, testing extraction rules, comparing SERP queries, or building a batch workflow, interactive mode is usually the fastest way to iterate before turning the final command into a script.

Startup flags (must precede the bare invocation, since they belong to the top-level scrapingbee command):

scrapingbee --keep-bg        # Keep your terminal's own background/theme colors instead of forcing black/light-grey
scrapingbee --no-drag-copy   # Disable click-drag-to-copy; use the Scroll/Select mouse-mode toggle instead

Every regular subcommand works inside the REPL exactly as it does on the shell — scrape, crawl, google, etc. — but with inline output, click-to-open local paths, and Ctrl+C cancellation that stops mid-request instead of waiting for the HTTP call to finish.

Slash Commands

Meta-commands are prefixed with : so they don't collide with API commands:

Click-drag across the scrollback (or in :view) selects text and copies it to your clipboard; the highlight stays visible until new output arrives. Dragging past the top or bottom edge auto-scrolls the selection. Plain clicks open underlined existing paths, including ./, ../, ~/, absolute paths, and paths with spaces. Mouse-wheel scrolling works across the scrollback and fixed status widgets. Launch with --no-drag-copy to instead use the Scroll/Select mouse-mode toggle — a fallback for terminals where drag-copy misbehaves. On Linux, clipboard copy needs a tool (wl-copy, xclip, or xsel) installed; macOS (pbcopy) and Windows (clip) work out of the box.

Key Bindings

Live Job Feedback

Long-running commands do not leave you staring at a frozen prompt. While a request, batch, crawl, or shell command is running, the REPL shows a live running line above the input, pins Ctrl+C to stop in the toolbar, and keeps output streaming into the scrollback. Batches and crawls show a compact progress widget; crawls also surface the current phase, URL, fetched count, saved count, and a honeycomb-style progress view when the terminal is large enough.

When a command finishes, the REPL adds a footer with success, failure, or stopped status plus duration. Non-zero shell commands and cancelled commands keep the command text available so you can edit and retry instead of rebuilding it from history.

The layout adapts as the terminal changes size: the banner collapses from full ASCII art to compact forms, toolbar fields paginate on narrow screens, and below the minimum usable size the UI shows a resize notice instead of accepting half-rendered commands.

Session Defaults

Use :set to apply a flag to every subsequent command — useful when you're working through a single country, proxy mode, or output format for a session:

:set country-code=fr
:set premium-proxy=true
scrape "https://example.com"        # runs with --country-code fr --premium-proxy true
scrape "https://example.com" --country-code us   # explicit flag wins
:unset premium-proxy

Inline flags always override session defaults. Type :show to see what's active.

API Key Inside the REPL

If no API key is configured at startup, the prompt flips to a masked API key: input. Running auth or logout inside the REPL re-prompts in-place rather than dropping back to the shell. auth --unsafe (the advanced features gate) must be run from a regular shell.

Overwriting Existing Files

When --output-file points at an existing path, the shell prompts for confirmation. Inside the REPL, pass --overwrite to skip the check:

scrape "https://example.com" --output-file result.html --overwrite

--overwrite works outside the REPL too — useful in CI/CD where there's no human to confirm.

scrape

The scrape command calls the HTML API to fetch any web page. CLI flags map 1:1 to API parameters (underscores become hyphens: render_js → --render-js). For predefined values like sort orders, both hyphens and underscores are accepted (e.g. --sort-by price-low and --sort-by price_low both work). Every boolean flag — including CLI-side ones like --verbose, --resume, --escalate-proxy — accepts an explicit true/false, matching the API parameter flags (--render-js true, --premium-proxy true, …). Bare --verbose still works. When a boolean is omitted, the CLI does not send it and the API default applies.

See the HTML API documentation — every code snippet includes a CLI tab with the equivalent command.

Basic Usage

Scrape a page and print HTML to stdout:

scrapingbee scrape "https://example.com"

Save output to a file (extension auto-detected):

scrapingbee scrape "https://example.com" --output-file result

Scrape with JavaScript rendering and premium proxy:

scrapingbee scrape "https://example.com" --render-js true --premium-proxy true

Extract data with AI:

scrapingbee scrape "https://example.com" --ai-query "extract the main article title and author"

Return page as markdown (great for LLM pipelines):

scrapingbee scrape "https://example.com" --return-page-markdown true

Scrape Parameters

Scrape flags correspond directly to HTML API parameters. The table below groups them by category — click any parameter name to see full documentation on the HTML API page.

Rendering

Proxy

Headers

Output Format

Screenshots

Extraction

Request

Configuration

Scraping Configurations

Use --scraping-config to apply a pre-saved configuration from your ScrapingBee dashboard. This lets you reuse commonly used settings without typing them each time.

scrapingbee scrape "https://example.com" --scraping-config "My-Config"

Inline options override configuration settings — so you can use a saved config as a base and customize individual parameters per request:

scrapingbee scrape "https://example.com" --scraping-config "My-Config" --premium-proxy false

You can also omit the scrape subcommand when invoking a saved config from the top level:

scrapingbee --scraping-config "My-Config" "https://example.com"

Create and manage configurations in the ScrapingBee request builder. Configuration names are case-sensitive and only accept alphanumeric characters, hyphens, and underscores.

Presets

Presets apply a predefined set of options. They only set flags you haven't already set, so you can override any preset value.

scrapingbee scrape "https://example.com" --preset screenshot --output-file page

CLI-Only Scrape Flags

These flags are specific to the CLI and do not have API parameter equivalents.

On 403 or 429 responses, automatically retry with premium proxy, then stealth proxy. Useful for sites with aggressive bot detection.

scrapingbee scrape "https://example.com" --escalate-proxy

Split text/markdown output into chunks of N characters for LLM or vector DB pipelines. Outputs NDJSON (one JSON object per chunk). Set to 0 to disable.

scrapingbee scrape "https://example.com" --return-page-markdown true --chunk-size 2000 --chunk-overlap 200

Number of overlapping characters between consecutive chunks. Only used when --chunk-size > 0.

Force the output file extension (e.g. html, json). Skips automatic extension inference when --output-file has no extension.

scrapingbee scrape "https://example.com" --output-file result --force-extension md

crawl

The crawl command follows links across pages using Scrapy under the hood. Three modes are available:

1. Quick crawl — start from URL(s), follow same-domain links:

scrapingbee crawl "https://example.com" --max-depth 2 --max-pages 50

2. Sitemap crawl — fetch all URLs from a sitemap:

scrapingbee crawl --from-sitemap "https://example.com/sitemap.xml" --max-pages 100

3. Project spider — run a Scrapy project spider with ScrapingBee middleware:

scrapingbee crawl my_spider --project ./my_scrapy_project

All scrape rendering, proxy, and extraction flags are also available for quick/sitemap crawls (e.g. --render-js, --premium-proxy, --ai-query). In project spider mode, set ScrapingBee API parameters directly in your ScrapingBeeRequest; CLI API flags are rejected there. Crawl also accepts -H/--header, --retries, --backoff, and --verbose. Client-side output/extraction flags such as --output-file, --smart-extract, --extract-field, and --fields are accepted for parity with other commands but do not affect crawl output; use API-side extraction (--extract-rules, --ai-query) or run export / jq after the crawl.

Quick Crawl

Start from one or more URLs and follow same-domain links. Each page is saved as a numbered file in the output directory, with a manifest.json mapping URLs to files.

scrapingbee crawl "https://docs.example.com" \
  --max-depth 3 \
  --max-pages 200 \
  --return-page-markdown true \
  --output-dir docs_crawl

Restrict crawling with URL patterns:

scrapingbee crawl "https://example.com" \
  --include-pattern "/blog/" \
  --exclude-pattern "/tag/" \
  --max-pages 50

Save only specific pages while crawling the full site for link discovery:

scrapingbee crawl "https://example.com" \
  --save-pattern "/product/" \
  --ai-query "extract the product name and price" \
  --max-pages 100

Sitemap Crawl

Fetch and parse a sitemap (including sitemap indexes) then crawl all discovered URLs:

scrapingbee crawl --from-sitemap "https://example.com/sitemap.xml" \
  --return-page-markdown true \
  --concurrency 20

Project Spider

Run any Scrapy spider from a project directory. ScrapingBee middleware and your API key are automatically injected:

scrapingbee crawl my_spider --project /path/to/scrapy/project --concurrency 10

In project spider mode, set ScrapingBee parameters in your spider's ScrapingBeeRequest (for example, params={"render_js": True}). CLI scrape/crawl API flags and --scraping-config are rejected. Infrastructure flags such as --concurrency, --download-delay, and --autothrottle still apply. You can use -p as a short alias for --project.

Non-HTML and Discovery-Phase Crawls

When the saved page response is non-HTML or extraction-focused (--screenshot-full-page true, --screenshot-selector, --extract-rules, --ai-extract-rules, --ai-query, --return-page-text, etc.), the crawler may need separate HTML requests to discover follow links. It runs a two-phase flow:

1. Discovery — cheap HTML-only requests build a pool of candidate URLs until the pool reaches --max-pages.
2. Save — one binary request per pooled URL, dispatched in priority order.

scrapingbee crawl "https://example.com" --screenshot-full-page true --max-pages 100

This targets exactly --max-pages saved files (with backfill on individual save failures), but discovery requests can add credits beyond the save requests. Screenshot crawls on link-rich sites are often much cheaper than paying for HTML and binary on every page, but the exact cost depends on how many discovery requests are needed.

For crawl options that return non-HTML responses (--extract-rules, --ai-query, --return-page-text, or screenshots without --json-response), the CLI prompts before running because discovery may require extra requests. Pass --confirm yes to skip that prompt in scripts.

Crawl Parameters

The positional argument — one or more URLs to start crawling from. In project spider mode, this is the spider name instead of a URL.

scrapingbee crawl "https://example.com"
scrapingbee crawl "https://example.com" "https://blog.example.com"
scrapingbee crawl my_spider --project ./my_project

Accepts a URL to a sitemap.xml file. The CLI fetches the sitemap (through the ScrapingBee API for proxy support), parses it (handling sitemap indexes recursively up to depth 2), and starts crawling all discovered page URLs.

scrapingbee crawl --from-sitemap "https://example.com/sitemap.xml"

Controls how many link-hops deep the crawler will follow from the start URLs. A depth of 0 means unlimited. Depth 1 means only pages directly linked from the start URLs.

scrapingbee crawl "https://example.com" --max-depth 2

Limits the number of pages saved to disk. A value of 0 means unlimited. Save failures (flaky 5xx errors, etc.) are backfilled from the URL pool so the limit reflects your effective output budget, not raw API responses.

In non-HTML or extraction-focused modes, the crawler can run a lightweight HTML discovery phase to collect URLs into a pool, then dispatch saves in priority order — see Non-HTML and Discovery-Phase Crawls.

scrapingbee crawl "https://example.com" --max-pages 100

When set, only pages whose URL matches this regex are saved to disk. All other pages are still visited for link discovery (using lightweight HTML-only requests) but their content is not saved. This lets you crawl an entire site for structure while only saving the pages you care about.

scrapingbee crawl "https://example.com" --save-pattern "/product/" --ai-query "extract product details"

When resuming a previous crawl, the CLI reads manifest.json in the output directory to skip already-crawled URLs and continue numbering files from where the previous run left off.

scrapingbee crawl "https://example.com" --output-dir my_crawl --resume

Auto-confirm interactive crawl prompts. Use this in CI/CD when you have already accounted for extra discovery requests:

scrapingbee crawl "https://example.com" --screenshot-full-page true --confirm yes

Requires advanced features setup. This feature executes shell commands and is disabled by default.

Run a shell command after the crawl finishes. The command receives $SCRAPINGBEE_OUTPUT_DIR. $SCRAPINGBEE_SUCCEEDED and $SCRAPINGBEE_FAILED are currently always 0 for crawl; inspect manifest.json / failures.txt if you need crawl result counts.

scrapingbee crawl "https://example.com" --on-complete "echo 'Done! Files in $SCRAPINGBEE_OUTPUT_DIR'"

Path to a Scrapy project directory for running project spiders. The CLI injects ScrapingBee middleware and your API key into the project's Scrapy settings automatically.

scrapingbee crawl my_spider --project /path/to/scrapy/project

Comma-separated list of domains the crawler is allowed to visit. By default, the crawler only follows links on the same domain as the start URL(s). Use this to explicitly whitelist additional domains.

scrapingbee crawl "https://example.com" --allowed-domains "example.com,blog.example.com"

Follow links to any domain, not just the start URL's domain. Use with caution — the crawl can expand rapidly. Combine with --max-pages to set a hard limit.

scrapingbee crawl "https://example.com" --allow-external-domains --max-pages 50

A regex pattern that URLs must match to be followed. Only links whose full URL matches this pattern will be visited. Useful for restricting crawls to specific sections of a site.

scrapingbee crawl "https://example.com" --include-pattern "/docs/" --max-pages 100

A regex pattern for URLs to skip. Links matching this pattern will not be followed, even if they match --include-pattern. Useful for avoiding pagination, tags, or other low-value pages.

scrapingbee crawl "https://example.com" --exclude-pattern "/tag/|/page/|/author/"

Delay in seconds between consecutive requests. Useful for being polite to the target server or avoiding rate limits. Accepts decimal values.

scrapingbee crawl "https://example.com" --download-delay 1.5

Enable Scrapy's AutoThrottle extension, which automatically adjusts the download delay based on the server's response time and load. Recommended for large crawls where you don't want to overwhelm the target.

scrapingbee crawl "https://example.com" --autothrottle --max-pages 500

Folder where crawl results are saved. Each page is written as a numbered file with a manifest.json mapping URLs to files. Defaults to crawl_<timestamp>.

scrapingbee crawl "https://example.com" --output-dir my_crawl

Maximum number of concurrent requests. Set to 0 (default) to auto-detect from your plan's concurrency limit. Higher values speed up crawls but use more credits in parallel. The CLI caps concurrency at min(--concurrency, --max-pages) to prevent overshoot.

scrapingbee crawl "https://example.com" --concurrency 20 --max-pages 100

Batch Processing

The --input-file flag enables batch mode on scrape, google, and all other scraper commands. Instead of processing a single item, the CLI reads a file of URLs (or queries, ASINs, etc.) and processes them concurrently.

Input

Batch input supports .txt (one URL per line) and .csv files. Use --input-column for CSV files:

# Text file (one URL per line)
scrapingbee scrape --input-file urls.txt

# CSV file with a "url" column
scrapingbee scrape --input-file sites.csv --input-column url

# Pipe from stdin
cat urls.txt | scrapingbee scrape --input-file -

Output

Results are saved as numbered files in the output directory (default: batch_<timestamp>):

scrapingbee scrape --input-file urls.txt --output-dir my_results

Alternative output formats:

# Single CSV file
scrapingbee google --input-file queries.txt --output-format csv --output-file results.csv

# NDJSON to stdout (great for piping)
scrapingbee scrape --input-file urls.txt --output-format ndjson | jq .title

Additional Options

Deduplication and Sampling

Clean up your input before spending credits. --deduplicate normalizes URLs (lowercases domains, strips fragments and trailing slashes) and removes duplicates. --sample picks N random items for testing your configuration before committing to a full run.

scrapingbee scrape --input-file urls.txt --deduplicate --sample 10

Post-Processing

Requires advanced features setup. This feature executes shell commands and is disabled by default.

Transform each result before it's written to disk by piping it through a shell command. The result body is sent to stdin, and the command's stdout replaces it. Works with any tool: jq for JSON filtering, sed for text manipulation, or custom scripts.

# Keep only the first 3 organic results from each Google search
scrapingbee google --input-file queries.txt --post-process "jq '.organic_results[:3]'"

# Extract just the title from each scraped page
scrapingbee scrape --input-file urls.txt --post-process "jq -r '.title // empty'"

Note: --post-process applies to files and ndjson output formats, but not to --update-csv.

Update CSV In-Place

Fetch fresh data for each row and add the results as new columns directly into the original CSV. The existing columns are preserved and new data is merged alongside them. Ideal for enriching datasets with live web data — prices, stock levels, ratings, or any extracted field.

scrapingbee scrape --input-file products.csv --input-column url \
  --extract-rules '{"price":".price","title":"h1"}' \
  --update-csv

The CLI reads the CSV, scrapes each URL in the specified column, flattens the JSON response, and writes the enriched CSV back. Nested JSON is automatically flattened to dot-notation columns (e.g. buybox.price).

Resume After Interruption

If a batch is interrupted (Ctrl+C, network issue, credit limit), re-run with --resume and the same --output-dir. The CLI scans existing output files and skips already-completed items, continuing from where it left off.

scrapingbee scrape --input-file urls.txt --output-dir my_batch --resume

Finding interrupted batches: run scrapingbee --resume with no other arguments (bare, at the top level) to scan the current directory for incomplete batches and print the exact resume command for each one, including the correct --output-dir:

scrapingbee --resume
# Found 1 incomplete batch(es):
#
#   [1] batch_20250101_120000/  —  42/100 complete, 58 remaining
#       scrapingbee scrape --input-file urls.txt --resume --output-dir batch_20250101_120000

Extract Specific Fields

Deprecated: --extract-field will be removed in v2.0.0. Use --smart-extract instead — same dot-path syntax plus auto-format detection, recursive search, filters, and more.

Pull values from JSON responses using dot-path notation. The output is one value per line, ready to pipe into another command or save as a list. If the path traverses an array, values from every item are extracted.

# Extract all URLs from Google search results
scrapingbee google "best laptops 2025" --extract-field organic_results.url

# Extract product ASINs from Amazon search, then fetch each product
scrapingbee amazon-search "headphones" --extract-field products.asin > asins.txt
scrapingbee amazon-product --input-file asins.txt --output-dir products

If the path doesn't match any data, the CLI prints a warning with all available dot-paths to help you find the correct one.

Run a Command After Completion

Requires advanced features setup. This feature executes shell commands and is disabled by default.

Trigger a notification, sync results to a database, or start a downstream pipeline when the batch finishes. The command receives environment variables with the results summary.

scrapingbee scrape --input-file urls.txt --on-complete "echo 'Done: $SCRAPINGBEE_SUCCEEDED ok, $SCRAPINGBEE_FAILED failed'"

Batch Parameters

Path to the file containing one item per line. Supports .txt and .csv formats. Use - to read from stdin. For CSV files, combine with --input-column to specify which column contains the target values.

Controls how batch results are written:

• individual files (default): Omit --output-format to write one file per result in the output directory, with a manifest.json index.
• csv: All results merged into a single CSV file. Use --output-file to write to a file, or omit it to stream CSV to stdout.
• ndjson: Newline-delimited JSON streamed to stdout. Use --output-file to write to a file.

When used with a CSV input file, fetches fresh data for each row and adds the results as new columns in the original CSV. Useful for enriching existing datasets.

scrapingbee scrape --input-file products.csv --input-column url \
  --extract-rules '{"price":".price"}' --update-csv

Requires advanced features setup.

Pipe each individual result through a shell command before writing to disk. The command receives the result body on stdin. Useful for filtering or transforming JSON.

scrapingbee google --input-file queries.txt --post-process "jq '.organic_results[:5]'"

When resuming a previous batch, the CLI scans the output directory for already-completed items and skips them. Numbering continues from the previous run.

Requires advanced features setup.

Shell command to run after batch completion. Your script receives $SCRAPINGBEE_OUTPUT_DIR for individual-file output, $SCRAPINGBEE_OUTPUT_FILE for CSV/NDJSON/update-CSV output, plus $SCRAPINGBEE_SUCCEEDED and $SCRAPINGBEE_FAILED — so it can process the output, trigger downstream workflows, or send alerts based on results.

For CSV input files, specifies which column contains the target values. Accepts a column name (from the header row) or a 0-based index. When omitted, the first column is used.

scrapingbee scrape --input-file sites.csv --input-column url
scrapingbee scrape --input-file data.csv --input-column 2

Folder where batch results are saved. Each result is written as a numbered file (e.g. 1.html, 2.json) with a manifest.json index mapping inputs to files. Defaults to batch_<timestamp>.

Write output to a specific file instead of stdout. For single-item commands (not batch), this saves the response directly. The file extension is auto-detected from the response type (HTML, JSON, PNG, etc.) unless you include one.

Overwrite an existing --output-file without prompting. Useful for CI/CD or repeatable scripts.

Maximum number of concurrent requests. Set to 0 (default) to auto-detect from your plan's concurrency limit via the usage API. Higher values speed up batch processing but use more credits simultaneously.

scrapingbee scrape --input-file urls.txt --concurrency 20

Normalize URLs and remove duplicates from the input before processing. URL normalization lowercases the domain, strips fragments, and removes trailing slashes. Useful when your input file may contain duplicate or near-duplicate URLs.

scrapingbee scrape --input-file urls.txt --deduplicate

Process only N random items from the input file. Useful for testing your batch configuration on a subset before running the full job. Set to 0 (default) to process all items.

scrapingbee scrape --input-file urls.txt --sample 10 --output-dir test_run

Suppress the per-item progress counter during batch processing. Useful when piping output or running in CI/CD where the progress updates would clutter logs.

Show HTTP status code, credit cost, resolved URL, and other response headers for each request. In verbose mode, the CLI displays exact credit costs for SERP commands (e.g. Credit Cost: 10) based on the request parameters.

Deprecated. Use --smart-extract instead.

Extract values from JSON responses using a dot-path expression, outputting one value per line. Supports nested paths and automatically iterates over arrays. The output is newline-separated, making it ideal for piping into --input-file of another command.

scrapingbee google "pizza" --extract-field organic_results.url
scrapingbee amazon-search "laptop" --extract-field products.asin > asins.txt

If the path doesn't match any data, the CLI prints a warning with all available dot-paths to help you find the correct one.

Deprecated. Use --smart-extract with '{name:path}' block syntax instead.

Filter JSON output to include only the specified comma-separated top-level keys. Useful for reducing output size when you only need certain parts of the response.

scrapingbee google "test" --fields "organic_results,meta_data"

Number of retry attempts on transient errors (HTTP 5xx, connection errors). Default is 3. Each retry uses exponential backoff controlled by --backoff.

Multiplier for exponential backoff between retries. Default is 2.0, meaning delays of 2s, 4s, 8s between retries. Lower values retry faster; higher values are gentler on the API.

Smart Extract

--smart-extract provides client-side data extraction using a path language. It auto-detects the response format — JSON, HTML, XML, CSV, NDJSON, Markdown, or plain text — and lets you drill into the data using a single expression. It works on single-item runs for scraper commands: scrape, google, fast-search, amazon-product, amazon-search, amazon-pricing, walmart-product, walmart-search, youtube-search, youtube-metadata, chatgpt, and gemini.

Replaces --extract-field and --fields, which are deprecated and will be removed in v2.0.0.

Batch mode: --smart-extract is applied in scrape batches when writing individual files (--output-dir) or CSV output (--output-format csv). It is not applied to --output-format ndjson, because the stream is written before extraction. On non-scrape commands (google, amazon-*, etc.), --smart-extract works for single-item runs only; use --extract-field (deprecated) or post-process with jq in batch until SERP batch support lands.

Quick Examples

# Extract top 3 Google result titles
scrapingbee google "query" --smart-extract 'organic_results[0:3].title'

# Extract email from HTML page (auto-detected)
scrapingbee scrape "https://example.com" --smart-extract '...a[href=*mailto*].text'

# Structured JSON output with named fields
scrapingbee scrape "https://example.com" --smart-extract '{titles:...h1},{links:...href[0:5]}'

# JSON schema mode (same format as --ai-extract-rules)
scrapingbee scrape "https://example.com" --smart-extract '{"titles": "...h1", "links": "...href[0:5]"}'

# Discover hidden API endpoints from XHR traffic
scrapingbee scrape "https://httpbin.scrapingbee.com/" --json-response true --render-js true \
  --wait 5000 --smart-extract 'xhr.body.paths[keys]'

Path Language Reference

Navigate — traverse into the structure

Select — pick specific items

Dict operations — inspect keys and values

Recursive search — find keys at any depth

Context expansion — go up in the tree (chainable anywhere)

Use ~N to see what surrounds a matched value. For example, find a price and go up 2 levels to see the full product card. Chain further after ~N — navigate into siblings, get keys, filter, or search recursively.

Filters — keep or exclude values

Key filters — filter dicts by a property value

Operators — combine multiple searches

Cannot mix | and & in one expression.

Important: Spaces around the operator are required — '...email | ...phone' works, but '...email|...phone' silently fails (the | is treated as part of the key name). This also applies to &.

Three Output Modes

1. Single path — raw values, one per line (default):

scrapingbee google "query" --smart-extract 'organic_results.title'
# ScrapingBee – The Best Web Scraping API
# ScrapingBee Python SDK
# ...

2. Block syntax — structured JSON with named fields:

scrapingbee google "query" --smart-extract '{titles:organic_results[0:3].title},{urls:organic_results[0:3].url}'
# {"titles": ["ScrapingBee...", ...], "urls": ["https://...", ...]}

3. JSON schema mode — same format as --ai-extract-rules:

scrapingbee google "query" --smart-extract '{"titles": "organic_results[0:3].title", "urls": "organic_results[0:3].url"}'
# {"titles": ["ScrapingBee...", ...], "urls": ["https://...", ...]}

Multiple fields in block syntax need separate {} pairs: '{a:path1},{b:path2}', not '{a:path1,b:path2}'.

Auto-Format Detection

The response format is detected automatically — no configuration needed:

Real-World Patterns

Extract all emails from a page:

scrapingbee scrape "https://example.com" --smart-extract '...a[href=*mailto*].text'

Find prices with regex:

scrapingbee scrape "https://example.com" --smart-extract '...text[=/\$\d+/]'

Get pricing section with context (for LLMs):

# Find price text, go up 2 levels to see the full product card
scrapingbee scrape "https://example.com" --smart-extract '...text[=/\$\d+/]~2'

Find a section by ID and extract its content:

scrapingbee scrape "https://example.com" --smart-extract '...*[id=faq]...text'

Exclude navigation and get only content text:

scrapingbee scrape "https://example.com" --smart-extract '...div[class!=nav][class!=footer]...text'

Sibling navigation (CSS h3 + p equivalent):

# Find an h3, go to parent, get the sibling paragraph
scrapingbee scrape "https://example.com" --smart-extract '...h3~1.p'

Discover hidden APIs from XHR traffic:

scrapingbee scrape "https://example.com" --json-response true --render-js true --wait 5000 \
  --smart-extract 'xhr.body.paths[keys]'

Drill through JSON strings (auto-parsed):

# xhr.body is a JSON string — smart-extract parses it automatically
scrapingbee scrape "https://httpbin.scrapingbee.com/" --json-response true --render-js true \
  --wait 5000 --smart-extract '{api:xhr.body.info.title},{endpoints:xhr.body.endpoints[keys][0:5]}'

Combine multiple searches with OR:

scrapingbee scrape "https://example.com" --smart-extract '...email | ...a[href=*mailto*].text'

Conditional output with AND (only if both found):

scrapingbee scrape "https://example.com" --smart-extract '...email & ...phone'

Extract from Google results with structured output:

scrapingbee google "web scraping" --smart-extract '{titles:organic_results[0:5].title},{urls:organic_results[0:5].url}'

Extract from YouTube search:

scrapingbee youtube-search "python tutorial" --type video \
  --smart-extract '{titles:results[0:3].title},{views:results[0:3].views}'

export

The export command merges numbered output files from a batch or crawl into a single file. It reads manifest.json (if present) to annotate each record with its source URL.

Examples

# Merge to NDJSON (default)
scrapingbee export --input-dir batch_20250101_120000 --output-file all.ndjson

# Merge to plain text
scrapingbee export --input-dir crawl_20250101 --format txt --output-file pages.txt

# Merge to CSV with flattened nested JSON
scrapingbee export --input-dir serps/ --format csv --flatten --output-file results.csv

# CSV with specific columns only
scrapingbee export --input-dir serps/ --format csv --columns "title,url,price" --output-file filtered.csv

# Deduplicate CSV rows
scrapingbee export --input-dir batch/ --format csv --deduplicate --output-file unique.csv

# Deeply nested CSV flattening
scrapingbee export --input-dir products/ --format csv --flatten --flatten-depth 8 --output-file products.csv

Export Parameters

• ndjson (default): One JSON object per line. If the source file is valid JSON, it's output as-is with an added _url field. Non-JSON files are wrapped as {"content": "...", "_url": "..."}.
• txt: Plain text output. Each file's content is separated by a blank line, prefixed with # URL when manifest is available.
• csv: Flattens JSON files into tabular rows. JSON arrays inside each file are expanded into individual rows. Use --flatten for nested objects and --columns to select specific fields.

In CSV mode, recursively flattens nested dictionaries to dot-notation column names. For example, {"buybox": {"price": 29.99}} becomes a column named buybox.price. Lists of dictionaries are indexed: buybox.0.price, buybox.1.price, etc.

By default, flattening stops at 5 nested levels and asks you to raise the depth for deeply nested data instead of silently producing partial columns.

In CSV mode with --flatten, sets the maximum nesting depth to expand. Increase this when the exporter reports that the detected nesting depth exceeds the default:

scrapingbee export --input-dir products/ --format csv --flatten --flatten-depth 8

The batch or crawl output directory to read from. The export command looks for numbered files (e.g. 1.json, 2.html) and optionally reads manifest.json to annotate each record with its source URL.

In CSV mode, remove duplicate rows from the output. Two rows are considered duplicates if all their column values are identical.

In CSV mode, include only the specified comma-separated column names. Rows missing all selected columns are dropped. Useful for extracting specific fields from large JSON responses.

scrapingbee export --input-dir results/ --format csv --columns "title,url,price" --output-file filtered.csv

Write the merged output to a file instead of stdout. The default outputs to stdout, which is useful for piping to other tools.

Overwrite an existing --output-file without prompting.

schedule

Requires advanced features setup. The schedule command executes shell commands via cron and is disabled by default.

The schedule command creates cron jobs to run any ScrapingBee CLI command at fixed intervals.

macOS/Linux only. schedule relies on cron, which isn't available on Windows — there it returns a clear "scheduling requires cron" message instead of registering a job.

Creating a Schedule

# Monitor a price every 5 minutes
scrapingbee schedule --every 5m --name btc-price \
  scrape "https://example.com/price" --extract-rules '{"price":".amount"}'

# Scrape news headlines every hour
scrapingbee schedule --every 1h --name news \
  google "breaking news" --search-type news

# Daily crawl
scrapingbee schedule --every 1d --name daily-crawl \
  crawl "https://example.com" --max-pages 50 --return-page-markdown true

Managing Schedules

# List all active schedules
scrapingbee schedule --list

# Stop a specific schedule
scrapingbee schedule --stop btc-price

# Stop all schedules
scrapingbee schedule --stop all

How It Works

The CLI uses your system's cron to run commands at the specified interval. Each schedule:

• Creates a cron entry tagged with the schedule name
• Logs output to ~/.config/scrapingbee-cli/logs/<name>.log
• Tracks metadata in ~/.config/scrapingbee-cli/schedules.json

Interval syntax: 5m (minutes), 1h (hours), 2d (days), or 60s and above. Seconds are rounded down to whole minutes, and the minimum interval is 1 minute.

Schedule Parameters

Duration string specifying how often to run the command. Uses cron under the hood:

• 5m → runs every 5 minutes (*/5 * * * *)
• 1h → runs every hour (0 */1 * * *)
• 2d → runs every 2 days (0 0 */2 * *)

Stop a schedule by name, removing its cron entry and registry record. Use --stop all to stop all active schedules.

scrapingbee schedule --stop btc-price

A human-readable name for the schedule. Used to identify it in --list output and to stop it with --stop. If omitted, a name is auto-generated from the command arguments.

scrapingbee schedule --every 1h --name hourly-news google "breaking news"

Display all active schedules in a table showing the name, interval, how long each has been running, and the full command. Useful for checking what's scheduled before adding or removing jobs.

scrapingbee schedule --list

Pipelines

The real power of the CLI emerges when you chain commands together. Every command is designed to compose — output from one step feeds naturally into the next. This turns the CLI into a data pipeline engine where web scraping is just the first stage.

Scrape to LLM: Building a Knowledge Base

Large language models and RAG (Retrieval-Augmented Generation) systems need clean text. The CLI can crawl an entire documentation site and convert every page to markdown — ready for embedding and indexing in a vector database.

scrapingbee crawl "https://docs.example.com" \
  --return-page-markdown true --max-pages 500 --output-dir knowledge_base

For single-page ingestion, use --chunk-size on the scrape command to split content into overlapping NDJSON chunks with metadata (URL, chunk index, total chunks, timestamp) — ready to pipe directly into an embedding API.

scrapingbee scrape "https://docs.example.com/guide" \
  --return-page-markdown true --chunk-size 2000 --chunk-overlap 200

Unix Piping: Composing with Standard Tools

The CLI speaks stdin and stdout fluently. Use --input-file - to read from a pipe and --output-format ndjson to stream structured results — connecting ScrapingBee to the entire Unix ecosystem.

Extract titles from a list of URLs and filter with jq:

cat urls.txt | scrapingbee scrape --input-file - \
  --output-format ndjson --extract-rules '{"title":"h1"}' | jq -r '.title'

Chain two ScrapingBee commands — search Google, then scrape the top results:

scrapingbee google "best python libraries 2025" \
  --extract-field organic_results.url | scrapingbee scrape --input-file - \
  --return-page-markdown true --output-dir articles

Data Enrichment: Augmenting Existing Datasets

Start with a CSV of products, competitors, or leads — and enrich it with live web data. The --update-csv flag adds scraped results as new columns directly into your existing file, preserving all original data.

scrapingbee scrape --input-file products.csv --input-column url \
  --extract-rules '{"price":".price","stock":".availability"}' --update-csv

This is particularly powerful for monitoring workflows: run it on a schedule and your CSV accumulates fresh data over time. Use --extract-rules to target exactly the fields you need — keeping your dataset clean and focused.

ETL: Extract, Transform, Load

For larger datasets, the batch → export → transform pattern gives you full control over each stage. Scrape in parallel, merge the results, then reshape into exactly the format your downstream system needs.

scrapingbee amazon-search --input-file queries.txt --output-dir raw_results
scrapingbee export --input-dir raw_results --format csv --flatten --output-file products.csv

The --flatten flag recursively expands nested JSON into dot-notation columns (buybox.price, seller.0.name), turning deeply nested API responses into flat CSV rows that work in any spreadsheet or database.

Monitoring: Scheduled Data Collection

Requires advanced features setup.

Combine schedule with any pipeline to run it automatically. The CLI registers a cron job that executes your command at the specified interval, with output logged for debugging.

scrapingbee schedule --every 1h --name competitor-prices \
  scrape --input-file competitors.csv --input-column url \
  --extract-rules '{"price":".price"}' --update-csv

Each run appends fresh data. Use --on-complete to trigger a notification, sync to a database, or kick off a downstream analysis when a batch job finishes.

scrapingbee schedule --every 6h --name news-digest \
  google --input-file queries.txt --output-dir news_results \
  --on-complete "python analyze.py"

Save-Pattern Crawling: Surgical Data Extraction

Sometimes you need to crawl an entire site for navigation structure but only extract data from specific pages. The --save-pattern flag crawls all pages for link discovery (using lightweight HTML requests) but only applies your expensive extraction options to pages whose URLs match the pattern.

scrapingbee crawl "https://store.example.com" \
  --save-pattern "/product/" --ai-query "extract product name, price, and reviews" \
  --max-pages 500

This can dramatically reduce API credit usage on large sites where only a fraction of pages contain the data you need.

Smart Extract: LLM-Optimized Extraction

Instead of feeding an entire HTML page or markdown document to your LLM, use --smart-extract to extract just the relevant section. Smaller context = better results, fewer tokens, lower cost.

# Extract only the article content, skipping nav and footer
scrapingbee scrape "https://example.com/article" \
  --smart-extract '...*[id=content]...text'

# Find a specific piece of data and get its surrounding context
scrapingbee scrape "https://example.com/pricing/" \
  --smart-extract '...text[=/\$\d+/]~3'

# Structured output ready for RAG ingestion
scrapingbee scrape "https://example.com" \
  --smart-extract '{"title": "...h1", "content": "...*[id=main]...text", "links": "...href[0:10]"}'

Smart extract works on any response format. Use it with --return-page-markdown true to extract from clean markdown, or with raw HTML for structure-aware extraction. See Smart Extract for the full path language reference.

Scraper API Commands

These commands wrap ScrapingBee's specialized scraper APIs. Full parameter documentation lives on each API's page — select the CLI tab for command-line examples.

All scraper commands support --input-file for batch processing and the same retry/output flags (--output-file, --retries, --backoff, --verbose, --tag). In batch mode, --output-format csv|ndjson streams merged results to stdout or to --output-file; omit --output-format to write individual files to --output-dir. --smart-extract works in single-item runs for all scraper commands, but batch extraction is currently limited to scrape as described in Smart Extract. Legacy flags --extract-field and --fields are also supported but deprecated.

Quick Examples

# Google search
scrapingbee google "web scraping best practices" --output-file results.json

# Fast search (lightweight, 10 credits per request)
scrapingbee fast-search "python web scraping"

# Amazon product
scrapingbee amazon-product B08N5WRWNW --output-file product.json

# Amazon pricing
scrapingbee amazon-pricing B08N5WRWNW --output-file pricing.json

# Amazon search
scrapingbee amazon-search "wireless headphones" --sort-by bestsellers

# Amazon pricing lookup
scrapingbee amazon-pricing B08N5WRWNW --domain com --country us

# Walmart product
scrapingbee walmart-product 123456789 --output-file product.json

# Walmart search
scrapingbee walmart-search "gaming laptop" --sort-by price-low

# YouTube search
scrapingbee youtube-search "python tutorial" --upload-date this-week

# YouTube metadata
scrapingbee youtube-metadata dQw4w9WgXcQ --output-file video.json

# YouTube metadata also accepts full video URLs
scrapingbee youtube-metadata "https://www.youtube.com/watch?v=dQw4w9WgXcQ"

# ChatGPT query
scrapingbee chatgpt "Summarize the latest AI news" --search true

# Gemini query
scrapingbee gemini "Summarize the latest AI news" --add-html true

Batch Examples

# Batch Google search
scrapingbee google --input-file queries.txt --output-format csv --output-file serps.csv

# Batch Amazon products
scrapingbee amazon-product --input-file asins.txt --output-dir products

Advanced Features

The --post-process, --on-complete, schedule, and REPL !command features execute arbitrary shell commands on your machine. To prevent accidental or unauthorized use, these are disabled by default and require explicit setup.

Why Are They Gated?

In AI agent environments, scraped web content could contain prompt injection attempts that trick an AI into constructing malicious shell commands. The exec gate ensures these features can only run when a human has deliberately enabled them.

How to Enable

Three conditions must be met before these features will run:

Step 1 — Set the environment variable:

export SCRAPINGBEE_ALLOW_EXEC=1

Add this to your ~/.bashrc or ~/.zshrc to persist across sessions.

Step 2 — Run the unsafe verification command:

scrapingbee auth --unsafe

This writes a verification flag to your config file (~/.config/scrapingbee-cli/.env).

Step 3 (optional) — Restrict allowed commands:

export SCRAPINGBEE_ALLOWED_COMMANDS="jq,head,python3 /path/to/transform.py"

Comma-separated list of allowed command prefixes. When set, only commands matching these prefixes can be executed by --post-process, --on-complete, schedule, or REPL !command. If not set, any command is allowed once the first two conditions are met.

Status and Audit

# Check if advanced features are enabled and view audit log
scrapingbee unsafe --list

# View recent shell command audit log
scrapingbee unsafe --audit

# View only the last N lines of the audit log
scrapingbee unsafe --audit --audit-lines 20

Disabling

To revoke advanced features:

scrapingbee logout

This removes both the API key and the unsafe verification flag. Alternatively, unset the environment variable (unset SCRAPINGBEE_ALLOW_EXEC).

To revoke only the unsafe flag while keeping your API key stored:

scrapingbee unsafe --disable

Utility Commands

Interactive step-by-step walkthrough of CLI features. 25 screens covering scraping, AI extraction, smart extract, screenshots, search APIs, crawling, batch processing, and export. Each step executes real API calls — expect to use approximately 100–150 credits over the full tutorial.

scrapingbee tutorial                    # Start from the beginning
scrapingbee tutorial --chapter 6        # Jump to Smart Extract chapter
scrapingbee tutorial --list             # List all chapters and steps
scrapingbee tutorial --reset            # Reset progress and start over
scrapingbee tutorial --output-dir out   # Save tutorial output files to a custom directory

Manage advanced features status and view the shell command audit log:

scrapingbee unsafe --list     # Check status
scrapingbee unsafe --audit    # View audit log

Filter the audit log by time range when needed:

scrapingbee unsafe --audit --audit-since 2026-03-31
scrapingbee unsafe --audit --audit-since 2026-03-31T14:00 --audit-until 2026-03-31T18:00

Print or open the ScrapingBee documentation URL:

scrapingbee docs           # Print the URL
scrapingbee docs --open    # Open in browser

scrapingbee --version

Check your API credit balance and plan concurrency. See Credits and Plan for details.

scrapingbee usage
scrapingbee usage --output-file usage.json

Save or display your API key. See Authentication for details.

scrapingbee auth

Remove stored API key and unsafe verification flag. If active schedules exist, logout warns and can stop them before removing the key. See Authentication for details.

scrapingbee logout

Get help for any command:

scrapingbee --help
scrapingbee scrape --help
scrapingbee crawl --help