如何在 selenium Python 中动态地一个一个地获取 Url？

Question

我是硒 Python 的新手。我想在谷歌上搜索一个关键字并打开它，在结果部分我想点击第一个 url 并获取数据然后返回点击第二个链接并获取数据......等等直到 10 个 URL。我已经在下面的代码中使用 x-path 完成了它，但我想动态地完成它而不编写一个链接的特定 x-path？PS - 我试过使用 for 循环，但我做不到。总而言之，我想在不指定特定 x 路径的情况下获得以下代码的结果，但为任何关键字动态获取 url。from selenium import webdriverfrom selenium.webdriver.common.keys import Keysfrom selenium.webdriver.common.by import Byimport timedriver=webdriver.Chrome(executable_path="E:\Sahil\selenium\chromedriver\chromedriver.exe")driver.get("https://www.google.com/")print(driver.title)driver.maximize_window()time.sleep(2)driver.find_element(By.XPATH, "//input[@name='q']").send_keys('selenium')driver.find_element(By.XPATH, "//div[@class='FPdoLc tfB0Bf']//input[@name='btnK']").send_keys(Keys.ENTER)# time.sleep(5)# 1>>>driver.find_element(By.PARTIAL_LINK_TEXT, "Selenium Web Driver").click()a=driver.find_elements(By.TAG_NAME, "p")for data in a:    print(data.text)driver.back()# 2>>>driver.find_element(By.PARTIAL_LINK_TEXT, "The Selenium Browser Automation Project :: Documentation ...").click()b=driver.find_elements(By.TAG_NAME, "p")for data in b:    print(data.text)driver.back()# 3>>>driver.find_element(By.PARTIAL_LINK_TEXT, "Selenium Tutorial for Beginners: Learn WebDriver in 7 Days").click()c=driver.find_elements(By.TAG_NAME, "p")for data in c:    print(data.text)driver.back()# 4>>>driver.find_element(By.PARTIAL_LINK_TEXT, "Selenium with Python — Selenium Python Bindings 2 ...").click()d=driver.find_elements(By.TAG_NAME, "p")for data in d:    print(data.text)driver.back()# 5>>>driver.find_element(By.PARTIAL_LINK_TEXT, "Selenium: Definition, How it works and Why you need it ...").click()e=driver.find_elements(By.TAG_NAME, "p")for data in e:    print(data.text)driver.back()

Answer 1

尝试这个 ：

from selenium import webdriver

from selenium.webdriver.common.keys import Keys

from selenium.webdriver.common.by import By

import time

driver=webdriver.Chrome("chromedriver.exe")

driver.get("https://www.google.com/")

print(driver.title)

driver.maximize_window()

time.sleep(2)

driver.find_element(By.XPATH, "//input[@name='q']").send_keys('selenium')

driver.find_element(By.XPATH, "//div[@class='FPdoLc tfB0Bf']//input[@name='btnK']").send_keys(Keys.ENTER)

a = driver.find_elements_by_xpath("//div[@class='r']/a") 

links = []

for x in a:      # this loop get all the webpages link and store into 'links' list.

    links.append(x.get_attribute('href'))

link_data = []

for new_url in links:    #go on every webpage and store page source in link_data list.

    print('new url : ' , new_url)

    driver.get(new_url)

    link_data.append(driver.page_source)

    driver.back()

#print('link data len : ' ,len(link_data)) 

#print('link data [0] : ' , link_data[0])  # print first webpage source.

此代码从所有链接获取所有数据并保存在link_data列表中。

对于 p 标签，您可以使用以下代码：

from bs4 import BeautifulSoup as bs

page = bs(link_data[0],'html.parser')

p_tag = page.find_all('p')

print(p_tag)