消息：元素 <div id="tabber_obj_0_div_3" class="ptr"> 无法滚动到视图中

Question

我正在尝试从给定的 URL 中抓取书籍的评论。我需要书名和每条评论在单独的行中。以下是我的代码。我同时使用硒和bs4。如果我不遵守帖子指南，任何反馈都将不胜感激并诚挚道歉，因为我对这个门户网站相对较新！

from selenium.webdriver import Firefox
browser = Firefox()
url ="https://www.booksaremagic.net/?q=h.tviewer&using_sb=status"
browser.get(url)
detailed_list = browser.find_element_by_xpath("//div[@title='Detailed']")
detailed_list.click()

books1 = []
for i in range(1,201):
    page_source1 = browser.page_source
    soup1 = BeautifulSoup(page_source1, "html.parser")
    for book1 in soup1.find_all('td',{"nbr ca"}):
        browser.find_element_by_tag_name('a[class="lightgrey bordered button moreinfo togsrus"]').click()
        browser.find_element_by_xpath("//*[@id='tabber_obj_0_div_3']").click()
        reviews = browser.find_element_by_id('tabber_obj_0_divbody_3').text
        all_books_scraped1 = [reviews]
        books1.append(all_books_scraped1)
        print("Finished Extracting Page:",i)
    try:
        select = Select(browser.find_element_by_class_name('quicknav')) 
        select.select_by_value(str(i))
    except:
        print("\nScraping Complete...")

输出应该是 df ：

(Cols)书评 (行)Book1 评论 1 Book1 评论 2 Book1 评论 N . . . Book9918 评论 N

Answer 1

我发现网站有某种 API 来获取数据，这是我的代码：

import requests as r
from bs4 import BeautifulSoup
import re

link = 'https://www.booksaremagic.net/?q=h.tviewer&using_sb=status&filt[status]=%20activenotcb%2C__instock&infscroll=1&&t=3&lim=500&dm=7&using_sb=status&overridelim=1&dm=7&offs=0'

# Get 500 books at once, if you're planning to get more, add 500 to the offs, so it goes like 0, 500, 1000, 1500, ...
# You can use up to 10000 at once, so 0, 10000, 20000, ...
res = r.get(link)
data = res.json()
books_5 = data["rs"]
# Each row in books_5 will contain html with 5 books in it
for row in books_5:
    for x in row.values():
        soup = BeautifulSoup(x, "html.parser")
        # Get the title
        title = soup.find_all('b')[0].text
        # get the link for the book
        link_book = soup.find_all('a')[0]["href"]
        # get the html page for the requested book
        res = r.get("https://www.booksaremagic.net/"+link_book+"&t=3")
        data = res.json()
        try:
            all_reviews_html = data["rs"][0]["sr"]["iv"]["rs"][0]["tabdata"][3]["tab_body"]
            soup_text = BeautifulSoup(all_reviews_html, "html.parser").text
            reviews = re.findall("“[^”]+”[^“]+", soup_text)
            # start and ends with " and the other is what follows after this —
            print(reviews)
        except:
            print(title, ": has no reviews.")
            pass
        # Here you have the title and all_reviews in html format for the book, extract your reviews with beautifulsoup
        # add your treatement here

您只需要从html代码all_reviews_html中提取评论，它只有评论，而不是整页。你有title。之后添加打印即可。