You can find elements by XPath selectors in Selenium by utilizing the find_element
and find_elements
methods and the By.XPATH
argument.
find_element
returns the first occurence of the XPath selector being used, while find_elements
returns all elements of the page that match the selector. And By.XPATH
simply tells Selenium to use the XPATH selector matching method.
Tip: //
in XPath matches an element wherever it is on the page whereas /
matches a direct child element.
Here is some example code which navigates to ScrapingBee website and extracts some data using XPath:
from selenium import webdriver
from selenium.webdriver.common.by import By
DRIVER_PATH = '/path/to/chromedriver'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
# Open Scrapingbee's website
driver.get("http://www.scrapingbee.com")
# Match all img tags on the page
all_img = driver.find_elements(By.XPATH, "//img")
# Match all a tags that contain the class of btn
all_btn = driver.find_elements(By.XPATH, "//a[contains(@class, 'btn')]")
# Match the first h1 tag on the page
first_h1 = driver.find_element(By.XPATH, "//h1")
# Get text of h1 tag
first_h1_text = first_h1.text
# Get count of all_img and all_btn
all_img_count = len(all_img)
all_btn_count = len(all_btn)
print(first_h1_text)
print(all_img_count)
print(all_btn_count)