如何通过selenium和python找到twitch上视频的href属性？

Question

我正在尝试查找特定用户的所有视频的 twitch 视频 ID。所以例如在这个页面上 https://www.twitch.tv/dyrus/videos/all

所以在这里我们链接了所有视频，但它并不是那么简单，只需抓取 html 并找到链接，因为它们似乎是动态生成的。

所以我听说了硒并做了这样的事情：

from selenium import webdriver

# Change path here obviously
driver = webdriver.Chrome('C:/Users/Jason/Downloads/chromedriver') 
driver.get('https://www.twitch.tv/dyrus/videos/all')
link_element = driver.find_elements_by_xpath("//*[@href]")


for link in link_element:
    print(link.get_attribute('href'))

driver.close()

这会在页面上返回一堆链接，但不是视频，我认为它们位于“更深”的地方，有什么意见吗？

提前致谢

Answer 1

我仍然建议以下几个更改：

• 始终以最大化模式打开Web浏览器，以便所有/大部分所需元素都在Viewport内。
• 如果您使用的是 Windows 操作系统，则需要在 WebDriver 变体名称的末尾附加扩展名 .exe，例如chromedriver.exe
• 当您识别元素时，请始终尝试在您的定位器策略中包含class 属性。
• 始终在 @Test 末尾调用 driver.quit() 以优雅地关闭和销毁 WebDriver 和 Web Client 实例。

• 这是您自己的代码块，经过上述调整：

  from selenium import webdriver
  from selenium.webdriver.chrome.options import Options
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC    
  
  options = Options()
  options.add_argument("start-maximized")
  options.add_argument("disable-infobars")
  driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\path\to\chromedriver.exe')
  driver.get('https://www.twitch.tv/dyrus/videos/all')
  link_elements = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.tw-interactive.tw-link[data-a-target='preview-card-image-link']")))
  for link in link_elements:
      print(link.get_attribute('href'))
  driver.quit()
  

• 控制台输出：

  https://www.twitch.tv/videos/295314690
  https://www.twitch.tv/videos/294901947
  https://www.twitch.tv/videos/294472813
  https://www.twitch.tv/videos/294075254
  https://www.twitch.tv/videos/293617036
  https://www.twitch.tv/videos/293236560
  https://www.twitch.tv/videos/292800601
  https://www.twitch.tv/videos/292409437
  https://www.twitch.tv/videos/292328170
  https://www.twitch.tv/videos/292032996
  https://www.twitch.tv/videos/291625563
  https://www.twitch.tv/videos/291192151
  https://www.twitch.tv/videos/290824842
  https://www.twitch.tv/videos/290434348
  https://www.twitch.tv/videos/290021370
  https://www.twitch.tv/videos/289561690
  https://www.twitch.tv/videos/289495488
  https://www.twitch.tv/videos/289138003
  https://www.twitch.tv/videos/289110429
  https://www.twitch.tv/videos/288804893
  https://www.twitch.tv/videos/288784992
  https://www.twitch.tv/videos/288687479
  https://www.twitch.tv/videos/288432438
  https://www.twitch.tv/videos/288117849
  https://www.twitch.tv/videos/288004968
  https://www.twitch.tv/videos/287689102
  https://www.twitch.tv/videos/287451192
  https://www.twitch.tv/videos/287267032
  https://www.twitch.tv/videos/287017431
  https://www.twitch.tv/videos/286819343
  

Answer 2

使用您的定位器，您将返回页面上包含href 属性的每个元素。你可以比这更具体一点，得到你正在寻找的东西。切换到 CSS 选择器...

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC    

# Change path here obviously
driver = webdriver.Chrome('C:/Users/Jason/Downloads/chromedriver') 
driver.get('https://www.twitch.tv/dyrus/videos/all')
links = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a[data-a-target='preview-card-image-link']")))

for link in links:
    print(link.get_attribute('href'))

driver.close()

从页面打印 40 个链接。