Many websites use infinite scrolling to load content as you scroll down the page instead of loading everything at once. This can reduce the amount of content loaded initially and improve the page's initial load time.
However, this makes scraping with PHP more difficult, because some of the content isn't available in the initial HTML. You need to scroll the page to trigger the loading of additional content before you can extract it.
In this tutorial, we'll see how to handle infinite scroll pagination with PHP using ScrapingBee's JavaScript Scenario. The scroll_y feature lets you scroll the page and trigger additional content to load.
Before you start, make sure PHP's cURL extension is installed and enabled. You can run php -m and check that curl appears in the output. Installation steps vary by operating system and PHP distribution.
Scraping the page without scrolling
We'll use this demo page for the tutorial. It initially displays some boxes and loads additional boxes as you scroll down. First, let's make a request without a JavaScript Scenario and see how many boxes we get in the response.
<?php
$ch = curl_init();
$BASE_URL = "https://app.scrapingbee.com/api/v1/?";
$API_KEY = "YOUR-API-KEY";
$parameters = array(
'url' => 'https://srujana-maddula.github.io/infinite-scroll-demo1/'
);
$query = http_build_query($parameters);
curl_setopt($ch, CURLOPT_URL, $BASE_URL.$query);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Authorization: Bearer ' . $API_KEY
));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($ch);
if (!$response) {
die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}
$httpStatus = curl_getinfo($ch, CURLINFO_HTTP_CODE);
echo 'HTTP Status Code: ' . $httpStatus . PHP_EOL;
if ($httpStatus !== 200) {
curl_close($ch);
die('Request failed with status code: ' . $httpStatus);
}
// This demo page marks each item with class="box", so counting how many
// times that string appears is a quick way to check the response.
// For real scraping tasks, use a proper parser like PHP's DOMDocument.
$boxCount = substr_count($response, 'class="box"');
echo 'Boxes found: ' . $boxCount . PHP_EOL;
$file = fopen("no_scrolling.html", "w");
fwrite($file, $response);
fclose($file);
?>
When you run this code, you'll see something like:
HTTP Status Code: 200
Boxes found: 9
This demo loads nine boxes initially. Production websites may load a different number of items.
Scraping with scroll_y
To load the additional content, we'll use a JavaScript Scenario. The scenario below scrolls down by 1,080 pixels, waits 500 milliseconds for the new content to load, and then repeats the process.
This demo page adds its boxes synchronously as soon as you scroll, so the wait isn't strictly necessary here. On real-world websites, though, scrolling often triggers an asynchronous network request that needs a moment to complete before the new content actually appears in the page. The wait instruction gives that request time to finish before ScrapingBee captures the page.
<?php
$ch = curl_init();
$BASE_URL = "https://app.scrapingbee.com/api/v1/?";
$API_KEY = "YOUR-API-KEY";
$js_scenario = ["instructions" => [
["scroll_y" => 1080],
["wait" => 500],
["scroll_y" => 1080],
["wait" => 500]
]];
$js_scenario = json_encode($js_scenario);
$parameters = array(
'url' => 'https://srujana-maddula.github.io/infinite-scroll-demo1/',
'js_scenario' => $js_scenario
);
$query = http_build_query($parameters);
curl_setopt($ch, CURLOPT_URL, $BASE_URL.$query);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Authorization: Bearer ' . $API_KEY
));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($ch);
if (!$response) {
die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}
$httpStatus = curl_getinfo($ch, CURLINFO_HTTP_CODE);
echo 'HTTP Status Code: ' . $httpStatus . PHP_EOL;
if ($httpStatus !== 200) {
curl_close($ch);
die('Request failed with status code: ' . $httpStatus);
}
//counts occurrences of class="box" in the raw HTML. Not a general parsing technique — use DOMDocument for real scraping.
$boxCount = substr_count($response, 'class="box"');
echo 'Boxes found: ' . $boxCount . PHP_EOL;
$file = fopen("y_scroll.html", "w");
fwrite($file, $response);
fclose($file);
?>
This time, the response is saved to y_scroll.html. When you run the code, you should see:
HTTP Status Code: 200
Boxes found: 18
We now have 18 boxes in the rendered HTML because the JavaScript Scenario scrolled far enough to trigger the next batch.
If you need to load even more content, you can add more scroll_y and wait instructions to the scenario. This works well when you have a rough idea of how far you need to scroll. If you don't know how much content a page has, ScrapingBee's infinite_scroll instruction keeps scrolling and waiting until the page stops loading new content or a limit you set is reached.