Yes!
Web scraping is a very useful skill to have in a world that operates with, and generates data in every second. Data is everywhere, and it is important to acquire the ability to easily extract it from online sources.
Without web scraping knowledge, it would very difficult to amass large amounts of data that can be used for analysis, visualization and prediction.
For example, without tools like
Requests
and BeautifulSoup, it would be very difficult to scrape Wikipedia's
S&P500 historical data
. We would have to manually copy and paste each data point from each page, which is very tedious.
However, thanks to these tools, we can easily scrape the historical data in milliseconds using this code:
import requests
from bs4 import BeautifulSoup as bs
url = "https://en.wikipedia.org/wiki/S%26P_500"
r = requests.get(url)
soup = bs(r.content, "html.parser")
for tr in soup.find("table", {"class": "wikitable"}).find_all("tr"):
print(" ".join([td.text for td in tr.find_all('td')]))
And as a result, we will get the historical data of S&P 500:
1970
0.10%
4.01%
$1.04
...