You can wait for the page to load in Puppeteer by using the waitForSelector method. This will pause execution until a specific element shows up on the page and indicates that the page has fully loaded. This feature is extremely helpful while performing web scraping on dynamic websites.
Here is some sample code that opens up ScrapingBee homepage and waits for the content section to show up:
const puppeteer = require('puppeteer');
async function waitForSelector() {
const browser = await puppeteer.launch({
headless: false
});
const page = await browser.newPage();
await page.goto("https://scrapingbee.com");
await page.waitForSelector('#content', { timeout: 5_000 });
// Do whatever you want with the page next
}
waitForSelector();
You can read more about the waitForSelector API in the
official docs
.