【发布时间】：2021-11-21 11:43:19
【问题描述】：
首先。我非常感谢@cruisepandey 在这个话题上帮助了我： How to crawl question and answer of Google People Also Ask with Selenium and Python?
所以我是这样使用他的代码的：
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.implicitly_wait(30)
wait = WebDriverWait(driver, 30)
driver.get("https://www.google.com/search?q=How%20to%20make%20bakery%3F&source=hp&ei=j0aZYYjRAvja2roPrcWcyAU&iflsig=ALs-wAMAAAAAYZlUn4NMUPjfIpQmrXSmjIDnaWjJXWIJ&ved=0ahUKEwjI1JDn0Kf0AhV4rVYBHa0iB1kQ4dUDCAc&uact=5&oq=How%20to%20make%20bakery%3F&gs_lcp=Cgdnd3Mtd2l6EAMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBNQAFgAYJMDaABwAHgAgAF-iAF-kgEDMC4xmAEAoAECoAEB&sclient=gws-wiz")
all_questions = driver.find_elements(By.XPATH, "//span[text()='People also ask']/../following-sibling::div/descendant::div[@data-hveid and @class and @jsname and @data-ved]")
print(len(all_questions))
j = 1
for question in all_questions:
time.sleep(1)
ele = driver.find_element(By.XPATH, f"(//span[text()='People also ask']/../following-sibling::div/descendant::div[@data-hveid and @class and @jsname and @data-ved])[{j}]")
j = j + 2
ele.click()
time.sleep(1)
answer = ele.find_element(By.XPATH, ".//../following-sibling::div").get_attribute('innerText')
print(answer)
print('--------------')
这段代码很有帮助。但我想问两个问题。
- 点击显示答案时。如果不使用 time.sleep(1)，我会使用 wait until answer show，如何准确获取 class 和 code to wait answer show？
- 如果网速慢，代码会出现问题。单击更多结果时不会显示。我尝试使用等到 invisibility_of_element_located 加载图标。但我发现 Xpath 不对。有什么办法吗？ 这是我对@cruisepandey 代码的代码更新：
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.implicitly_wait(30)
wait = WebDriverWait(driver, 30)
driver.get("https://www.google.com/search?q=How%20to%20make%20bakery%3F&source=hp&ei=j0aZYYjRAvja2roPrcWcyAU&iflsig=ALs-wAMAAAAAYZlUn4NMUPjfIpQmrXSmjIDnaWjJXWIJ&ved=0ahUKEwjI1JDn0Kf0AhV4rVYBHa0iB1kQ4dUDCAc&uact=5&oq=How%20to%20make%20bakery%3F&gs_lcp=Cgdnd3Mtd2l6EAMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBMyBAgAEBNQAFgAYJMDaABwAHgAgAF-iAF-kgEDMC4xmAEAoAECoAEB&sclient=gws-wiz")
all_questions = driver.find_elements(By.XPATH, "//span[text()='People also ask']/../following-sibling::div/descendant::div[@data-hveid and @class and @jsname and @data-ved]")
print(len(all_questions))
j = 1
for question in all_questions:
time.sleep(1)
ele = driver.find_element(By.XPATH, f"(//span[text()='People also ask']/../following-sibling::div/descendant::div[@data-hveid and @class and @jsname and @data-ved])[{j}]")
j = j + 2
ele.click()
# Question 1: To waiting answer show.
timeout = 30
answer_css_class = 'wWOJcd'
try:
is_question_show = WebDriverWait(driver, timeout).until(
EC.presence_of_element_located((By.CLASS_NAME, answer_css_class))
)
except TimeoutException:
pass
time.sleep(1)
answer = ele.find_element(By.XPATH, ".//../following-sibling::div").get_attribute('innerText')
# Question 2: To waiting G loading icon when click more answer with slow internet.
loading_element_xpath = '/html/body/div[7]/div/div[9]/div[1]/div/div[2]/div[2]/div/div/div[2]/div/div/div[1]/g-loading-icon'
try:
is_question_show = WebDriverWait(driver, timeout).until(
EC.presence_of_element_located((By.XPATH, loading_element_xpath))
)
except TimeoutException:
pass
print(answer)
print('--------------')
那么有什么办法不使用 time.sleep 并使用 wait until in selenium？
【问题讨论】：
标签： python selenium selenium-webdriver xpath wait