我无法使用 selenium 或 BeautifulSoup 或两者结合用于动态网站返回评论文本

Question

我正在尝试从动态网站 (page) 中抓取评论。 我尝试了以下代码，但没有一个返回评论。

使用 BeautifulSoup：

my_url = 'same above link'
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()
page_soup = bs(page_html, "html.parser")
revs = page_soup.findAll("div", {"id":"product-comment-list"})

print(revs)

此代码返回：[]

使用硒：

chrome_path = r"C:\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("same link above")

elm_revs = driver.find_element_by_xpath('//a[@data-tab-name="comments"]')

driver.implicitly_wait(5)

driver.execute_script("arguments[0].click();", elm_revs)


try:
    revs = WebDriverWait(driver, 10).until(
        EC.presence_of_all_elements_located((By.ID, "product-comment-list"))
    )
finally:
    driver.quit()

print(revs.text)

time.sleep(5)

driver.quit()

此代码返回：urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=52205): Max retries exceeded with url: /session/63eeb4529f25314429dc7bf2926e010a/element/2ba52c02-56e8-4de8-8432-5e9148c2fcec/text (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000018BE37DE370>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))

或者如果我使用

try:
    revs = WebDriverWait(driver, 10).until(
        EC.presence_of_all_elements_located((By.ID, "product-comment-list"))
    )
finally:
    print(revs.text)

它什么也没带回来。只是空白输出！

结合使用 BeautifulSoup 和 Selenium：

chrome_path = r"C:\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("same above link")


elm_revs = driver.find_element_by_xpath('//a[@data-tab-name="comments"]')
page = driver.execute_script("arguments[0].click();", elm_revs)


html = driver.page_source
page_soup = bs(html, "html.parser")

for tag in page_soup.findAll('id')
    print(tag.text)

这个返回：[]

Answer 1

两个问题，您只有一个产品评论并使用 .text 打印元素。

revs = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "product-comment-list")))

print(revs.text)