Python

Search

how to scrape multiple pages using selenium in python

# importing necessary packages
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
  
# for holding the resultant list
element_list = []
  
for page in range(1, 3, 1):
    
    page_url = "https://webscraper.io/test-sites/e-commerce/static/computers/laptops?page=" + str(page)
    driver = webdriver.Chrome(ChromeDriverManager().install())
    driver.get(page_url)
    title = driver.find_elements_by_class_name("title")
    price = driver.find_elements_by_class_name("price")
    description = driver.find_elements_by_class_name("description")
    rating = driver.find_elements_by_class_name("ratings")
  
    for i in range(len(title)):
        element_list.append([title[i].text, price[i].text, description[i].text, rating[i].text])
  
print(element_list)
  
#closing the driver
driver.close()

Comment

PREVIOUS	NEXT

Code Example
Python :: Display if the column(s) contain duplicates in the DataFrame
Python :: decode multipart/form-data python
Python :: sqlite3 python
Python :: square root python
Python :: python bit shift by 3
Python :: __new__ python
Python :: pygame.draw.rect()
Python :: python aes encryption
Python :: telethon send image
Python :: python bar plot groupby
Python :: pygame tutorial
Python :: what is self in python
Python :: get the name of a current script in python
Python :: display data from database in django
Python :: python tic tac toe
Python :: breadth first search python
Python :: python print trailing zeros
Python :: read excel spark
Python :: Python NumPy split Function Example
Python :: legend font size python matplotlib
Python :: how to run python program in sublime text 3 windows
Python :: UTC to ISO 8601 with TimeZone information (Python 3):
Python :: Determine the sum of al digits of n
Python :: Convert two lists into a dictionary in python
Python :: discord python tts
Python :: save numpy array
Python :: how to show a frequency distribution based on date in python
Python :: pretty size python
Python :: pydrive upload file to folder
Python :: python fill zeros left

Search

PYTHON

how to scrape multiple pages using selenium in python

ADD CONTENT