Effortlessly extract data with our AI-powered web scraping API. Simplify data extraction, get clean JSON outputs and adapt to page changes. Try it free today! Use our AI-powered web scraper to extract any data from the internet at scale with ease. Use our AI Web Scraping API to effortlessly extract details on competitor product information, consumer reviews, and pricing data from a wide array of e-commerce sites. Intelligently detect and extract the relevant email addresses and contact information to help you scale your outbound marketing efforts. Use our advanced AI web scraping API to aggregate and summarise news articles from various sources, delivering concise, relevant insights tailored to your use case. AI Web Scraping made easy, give our quick start code a spin by grabbing your free API key. Input parameters Describe here in natural language the content you want to extract. Narrow down the page content, by optionally providing a specific DOM entry as root element. JSON rules specifying what the output of each scraped JSON object should be. Input Output Input Output Wondering how our customers use our web scraping API? ScrapingBee web scraping API is designed to extract data from any website, including those with advanced anti-bot technology. It utilizes stealth/premium proxies and cutting-edge headless browser technology to prevent being blocked. Getting HTML is cool, but getting formatted JSON data is better. Thanks to our easy-to-use extraction rules, get just the data you need with one simple API call. If you need to click, scroll, wait for some elements to appear, or just run some custom JavaScript code on the website you want to scrape, check our JS scenario feature. Need a screenshot of that website and not HTML? You can do this very easily with our screenshot feature. We also support full-page and partial screenshots. Scraping search engine result pages is extremely painful because of rate limits. Thanks to our Google search API, it's now easier than ever. If coding is not your thing, you can leverage our Make integration to create custom web scraping engines without writing a single line of code. Cancel anytime, no questions asked! Need more credits and concurrency per month? Not sure what plan you need? Try ScrapingBee with 1000 free API calls. (No credit card required) Check out our documentation for more on how to utilise our API for your needs.AI Web Scraping API
based on 100+ reviews.
Common AI Web Scraping Use Cases
Easily scrape e-commerce data
Extract emails and contact info
Summarise and aggregate news content
Quickly get started scraping the internet with AI
ai_query
ai_selector
ai_extract_rules
# Install the Python ScrapingBee library:
# `pip install scrapingbee`
from scrapingbee import ScrapingBeeClient
import json
client = ScrapingBeeClient(api_key='SIGN_UP_TO_GET_YOUR_FREE_API_KEY')
response = client.get(
'https://www.amazon.com/s?k=dslr+camera',
params={
'ai_query': 'Return a list of products and their prices and a link to the product page',
'ai_extract_rules': json.dumps({
"product name": {
'type': 'list',
'description': 'the full name of the product verbatim from the page',
},
"product price": {
'type': 'number',
'description': 'price of the product',
},
"link to product page": {
'type': 'string',
'description': 'the url that links to the products page',
}
})
}
)
print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content)
{"product name": ["Panasonic LUMIX GH7 Mirrorless Micro Four Thirds Camera with Enhanced Video Speed and Quality, Optimized Workflows from Shooting to Post-Production, Adobe Cloud Compatible & Real-Time LUT, DC-GH7BODY", "Panasonic LUMIX G100 4k Mirrorless Camera for Photo and Video, Built-in Microphone with Tracking, Micro Four Thirds Interchangeable Lens System, 12-32mm Lens, 5-Axis Hybrid I.S., DC-G100DKK (Black)", "Panasonic LUMIX G9II Micro Four Thirds Camera, 25.2MP Sensor with Phase Hybrid AF, Powerful Image Stabilization, High-Speed Perfomance and Mobility, Flagship Model of G Series - DC-G9M2BODY", "Canon EOS Rebel T7 DSLR Camera with 18-55mm Lens | Built-in Wi-Fi | 24.1 MP CMOS Sensor | DIGIC 4+ Image Processor and Ful...", "Canon EOS Rebel T7 DSLR Camera Bundle w/ Canon EF-S 18-55mm f/3.5-5.6 is II Lens + 2pc SanDisk 64GB Memory Cards, Wide Ang...", "Canon EOS Rebel T100 (EOS 4000D) DSLR Camera w/EF-S 18-55mm F/3.5-5.6 Zoom Lens + 64GB Memory Card, Case, Hood, Grip-Pod, ...", "Canon EOS 4000D / Rebel T100 DSLR Camera w/EF-S 18-55mm F/3.5-5.6 Zoom Lens + 64GB Memory, Filters,Case, Tripod, Flash, an...", "Canon EOS 4000D / Rebel T100 DSLR Camera with EF-S 18-55mm Zoom Lens + SanDisk 128GB Memory Card + Tripod + Case + Wideang...", "Canon EOS Rebel T7 DSLR Camera with 18-55mm is II Lens + Canon EF 75-300mm f/4-5.6 III Lens and 500mm Preset Lens + 32GB M...", "Canon EOS 2000D / Rebel T7 DSLR Camera with EF-S 18-55mm Lens + SanDisk 32GB Card Tripod Case Wideangle Lenses ZeeTech Acc...", "Panasonic Lumix G7 4K Digital Mirrorless Camera Bundle with Lumix G Vario 14-42mm and 45-150mm Lenses, 16MP, 3-Inch Touch ...", "Canon EOS R100 Mirrorless Camera RF-S18-45mm F4.5-6.3 is STM & RF-S55-210mm F5-7.1 is STM Lens Kit, 24.1 Megapixel CMOS (A...", "Panasonic LUMIX FZ80D Compact Camera with 20-1200mm Zoom Lens, Point and Shoot Digital Camera with 4K 2000D / Rebel T7 DSLR Camera with EF-S 18-55mm Lens + SanDisk 32GB Card Tripod Case Wideangle Lenses ZeeTech Acc...", "Panasonic Lumix G7 4K Digital Mirrorless Camera Bundle with Lumix G Vario 14-42mm and 45-150mm Lenses, 16MP, 3-Inch Touch ...", "Canon EOS R100 Mirrorless Camera RF-S18-45mm F4.5-6.3 is STM & RF-S55-210mm F5-7.1 is STM Lens Kit, 24.1 Megapixel CMOS (A...", "Panasonic LUMIX FZ80D Compact Camera with 20-1200mm Zoom Lens, Point and Shoot Digital Camera with 4K Video/Photo Recordin...", "Nikon D7500 20.9MP DSLR Camera with AF-S DX NIKKOR 18-140mm f/3.5-5.6G ED VR Lens, Black", "G-Anica Digital Camera, 48MP Cameras for Photography with WiFi & App Control, 4K Vlogging Camera for YouTube,Compact Camer...", "KODAK PIXPRO AZ405-BK 20MP Digital Camera 40X Optical Zoom 24mm Wide AngVideo/Photo Recordin...", "Nikon D7500 20.9MP DSLR Camera with AF-S DX NIKKOR 18-140mm f/3.5-5.6G ED VR Lens, Black", "G-Anica Digital Camera, 48MP Cameras for Photography with WiFi & App Control, 4K Vlogging Camera for YouTube,Compact Camer...", "KODAK PIXPRO AZ405-BK 20MP Digital Camera 40X Optical Zoom 24mm Wide Angle Lens Optical Image Stabilization 1080P Full HD ...", "Canon EOS 2000D / Rebel T7 DSLR Camera with EF-S 18-55mm Lens + SanDisk 32GB Card Tripod Case Wideangle Lenses ZeeTech Acc...Bundle", "Canon EOS 4000D / Rebel T100 DSLR Camera 18-55mm Zoom Lens + ZeeTech Accessory Bundle with SanDisk 32GB Memory Card, Bag, ...", "Nikon D3100 14.2MP DSLR Camera with AF-S DX 18-55mm f/3.5-5.6 VR Zoom Lens - (Black) (Discontinued) (Renewed)", "Panasonic LUMIX G9II Micro Four Thirds Camera, 25.2MP Sensor with Phase Hybrid AF, Powerful Image Stabilization, High-Speed Perfomance and Mobility, Flagship Model of G Series - DC-G9M2BODY Bundle", "Saneen Digital Camera, 4k Cameras for Photography & Video, 64MP WiFi Touch Screen Vlogging Camera for YouTube with Flash, ...", "Panasonic Lumix G7 4K Digital Mirrorl"Nikon D3100 14.2MP DSLR Camera with AF-S DX 18-55mm f/3.5-5.6 VR Zoom Lens - (Black) (Discontinued) (Renewed)", "Panasonic LUMIX G9II Micro Four Thirds Camera, 25.2MP Sensor with Phase Hybrid AF, Powerful Image Stabilization, High-Speed Perfomance and Mobility, Flagship Model of G Series - DC-G9M2BODY Bundle", "Saneen Digital Camera, 4k Cameras for Photography & Video, 64MP WiFi Touch Screen Vlogging Camera for YouTube with Flash, ...", "Panasonic Lumix G7 4K Digital Mirrorl Digital Camera, 4k Cameras for Photography & Video, 64MP WiFi Touch Screen Vlogging Camera for YouTube with Flash, ...", "Panasonic Lumix G7 4K Digital Mirrorless Camera Bundle with Lumix G Vario 14-42mm and 45-150mm Lenses, 16MP, 3-Inch Touch LCD, DMC-G7WK (USA Black) Bundle"], "product price": [479, 649, null, 479, ess Camera Bundle with Lumix G Vario 14-42mm and 45-150mm Lenses, 16MP, 3-Inch Touch LCD, DMC-G7WK (USA Black) Bundle"], "product price": [479, 649, null, 479, 649, 414, 429, 439, 739, 449.99, 597.99, 499, 447.99, 996.95, 125.60, 179.99, 399.99, 280.00, 429.99], "link to product page": ["/Canon-Rebel-T7-18-55mm-II/dp/B649, 414, 429, 439, 739, 449.99, 597.99, 499, 447.99, 996.95, 125.60, 179.99, 399.99, 280.00, 429.99], "link to product page": ["/Canon-Rebel-T7-18-55mm-II/dp/B07C2Z21X5/ref=sr_1_1", "/Panasonic-LUMIX-GH7/dp/B0D613NW2M/ref=sr_1_1", "/Panasonic-LUMIX-G100/dp/B0CQCDTT82/ref=sr_1_2", "/Canon-Rebel-T7-18-55mm-II/dp/B07C2Z21X5/ref=sr_1_1", "/Canon-T6-Digital-Telephoto-Accessory/dp/B01D93Z89W/ref=sr_1_2", "/Canon-18-55mm-3-5-5-6-Grip-Pod-Professional/dp/B0C7WJ9B5B/ref=sr_1_3", "/Canon-18-55mm-3-5-5-6-Filters-Renewed/dp/B0CFSBSRMY/ref=sr_1_4", "/Canon-18-55mm-75-300mm-Filters-Professional/dp/B08BFHJX1J/ref=sr_1_5", "/G-Anica-Photography-Entry-Level-Beginners-Lens-32GB/dp/B0DF78VSTZ/ref=sr_1_9", "/Canon-EOS-2000D-Rebel-T7/dp/B0BHJNWD1Z/ref=sr_1_11", "/Panasonic-Mirrorless-Digital-14-42mm-45-150mm/dp/B07887SCFK/ref=sr_1_12", "/Canon-RF-S18-45mm-RF-S55-210mm-Mirrorless-Lightweight/dp/B0C5P59T9J/ref=sr_1_13", "/Panasonic-Compact-20-1200mm-Recording-Stabilizer/dp/B0D8239188/ref=sr_1_14", "/Nikon-20-9MP-Camera-18-140mm-3-5-5-6G/dp/B06Y5RTN1T/ref=sr_1_8", "/Nikon-D3100-18-55mm-Discontinued-Manufacturer/dp/B01CO0XNG6/ref=sr_1_15", "/Canon-18-55mm-SanDisk-Backpack-Accessory/dp/B0B5YMYTQM/ref=sr_1_16", "/Saneen-Digital-Cameras-Photography-Vlogging/dp/B0CFZRPQMZ/ref=sxbs_sbv_search_btf"]}
Six ways to use ScrapingBee for web harvesting
From a general web scrape to JavaScript rendering, our simple API does it all.
Bypass Anti-Bot Technology
Data Extraction
JavaScript scenario
Screenshots
Search Engine Result Page
No code web scraping
Simple, transparent pricing.
Highly Versatile and Scalable Web Scraping API
Frequently Asked Questions
Related Articles
Scrape content easily by typing your scraping prompt in everyday language.
No DOM analysis or selector updates necessary; our AI automatically adjusts to page layout changes.
Receive output in a clean, structured JSON format.
Bypass anti-bot technology with our industry leading web scraping API.
Searches
API Credits
Concurrent requests
Geotargeting
Screenshots, Extraction Rules, Google Search API
Priority Email Support
Dedicated Account Manager
Team Management
All prices are exclusive of VAT.
AI web scraping requests cost an additional five credits on top of the regular API cost.
Register your ScrapingBee account at
https://app.scrapingbee.com/account/register
to get your 1,000 free API calls and check out the
documentation of the AI Web Scraping API
Yes, our AI web scraping functionality is a complementary feature to our existing highly scalable scraping platform that’s able to process millions of requests a day.
You can contact us either via the contact form on our website or our live chat where you can talk to one of our customer support agents who are scraping experts in their own right. We can help you to scale your scraping project with ease.
Yes, our AI web scraping support is built on top of our existing platform and can make full use of its broad JavaScript rendering support.
