【问题标题】:BeautifulSoup module isn't detecting any tagsBeautifulSoup 模块未检测到任何标签
【发布时间】:2019-07-09 23:20:12
【问题描述】:

我是 python 的新手,正在尝试做一个项目,我在亚马逊产品页面上打开所有评论链接。为什么soup.select() 方法不能为 python 链接找到正确的 html 标记

import webbrowser, requests, sys, bs4, logging

logging.basicConfig(level=logging.DEBUG, format=' %(asctime)s - % (levelname)s - %(message)s')
print("Searching...") # Text to display while searching amazon

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) 
        AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 
        Safari/537.36'}
url = input("Enter the url: ")
res = requests.get(url, headers=headers)
res.raise_for_status()

# Retrieve reviews found
soup = bs4.BeautifulSoup(res.text, features='html.parser')

# Open a tab for each review found
linkElems = soup.select('div.a-row a.a-size-base.a-link-normal.review- 
    title.a-color-base.a-text-bold')
numOpen = min(5, len(linkElems))
logging.debug(linkElems)
for i in range(numOpen):
    logging.debug("Link is: " + str(linkElems[i].get('href')))
    webbrowser.open('https://amazon.com' + linkElems[i].get('href'))

我希望这段代码能够生成并打开产品评论链接列表,但是当我运行它时,找到的标签列表为空。

【问题讨论】:

    标签: python python-3.x web-scraping beautifulsoup html-parsing


    【解决方案1】:

    更新 1: 当 op 编辑他的帖子并以他的代码格式修复空格时,我正在更新我的答案。

    当程序请求链接时,写下(或粘贴)您的链接并在按回车之前添加一个空格

    输入末尾的多余空格将阻止 IDE 在浏览器窗口中打开链接,而不是使用返回键结束输入,因此它将按预期继续执行输入函数后的下一个代码。

    这样做,您的代码实际上可以正常工作,正如我在第一个答案中所展示的那样。

    我之前的回答: 我注意到你在这里有不正确的空格'a-color-base'

    替换这一行:

    linkElems = soup.select('div.a-row a.a-size-base.a-link-normal.review-title.a- color-base.a-text-bold')
    

    linkElems = soup.select('div.a-row a.a-size-base.a-link-normal.review-title.a-color-base.a-text-bold')
    

    额外: 目前您的代码也仅适用于 amazon.com ,要使您的代码适用于其他亚马逊网站,如 amazon.in amazon.co.uk,您需要修改此行:

    webbrowser.open('https://amazon.com' + linkElems[i].get('href'))
    

    类似于:

    from urllib.parse import urlparse
    url_components = urlparse(url)
    webbrowser.open('https://' + url_components.netloc + linkElems[i].get('href'))
    

    现在它可以与 amazon.in 等其他亚马逊网站正常工作,而不仅仅是 amazon.com,试试吧。

    试驾:

    Enter the url: https://www.amazon.in/Intex-PB-16K-Poly-16000mAH-Lithium/dp/B07843GH8X/ref=cm_cr_srp_d_product_top?ie=UTF8 
     2019-02-15 22:51:28,940 - DEBUG - Starting new HTTPS connection (1): www.amazon.in:443
     2019-02-15 22:51:30,125 - DEBUG - https://www.amazon.in:443 "GET /Intex-PB-16K-Poly-16000mAH-Lithium/dp/B07843GH8X/ref=cm_cr_srp_d_product_top?ie=UTF8%20 HTTP/1.1" 200 None
     2019-02-15 22:51:32,019 - DEBUG - [<a class="a-size-base a-link-normal review-title a-color-base a-text-bold" data-hook="review-title" href="/gp/customer-reviews/R123ICSCDM2MF3/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&amp;ASIN=B07843GH8X">Nice product.</a>, <a class="a-size-base a-link-normal review-title a-color-base a-text-bold" data-hook="review-title" href="/gp/customer-reviews/R3OTWE19SMPPVQ/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&amp;ASIN=B07843GH8X">Not a good product. It takes a day to charge the ...</a>, <a class="a-size-base a-link-normal review-title a-color-base a-text-bold" data-hook="review-title" href="/gp/customer-reviews/R1Z51ERFCD7D6P/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&amp;ASIN=B07843GH8X">not portable easily..</a>, <a class="a-size-base a-link-normal review-title a-color-base a-text-bold" data-hook="review-title" href="/gp/customer-reviews/R24GYC4HRBGTM1/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&amp;ASIN=B07843GH8X">Five Stars</a>, <a class="a-size-base a-link-normal review-title a-color-base a-text-bold" data-hook="review-title" href="/gp/customer-reviews/R2P5ILE8KQF8PJ/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&amp;ASIN=B07843GH8X">Fine not superb</a>, <a class="a-size-base a-link-normal review-title a-color-base a-text-bold" data-hook="review-title" href="/gp/customer-reviews/R3C9ZYFRT9NWAK/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&amp;ASIN=B07843GH8X">Quality &amp; Performance</a>, <a class="a-size-base a-link-normal review-title a-color-base a-text-bold" data-hook="review-title" href="/gp/customer-reviews/R16F4OE3LWHHQI/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&amp;ASIN=B07843GH8X">Worst experience. Don’t buy</a>, <a class="a-size-base a-link-normal review-title a-color-base a-text-bold" data-hook="review-title" href="/gp/customer-reviews/R26ROATK8PU6TL/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&amp;ASIN=B07843GH8X">Very bad product</a>]
     2019-02-15 22:51:32,019 - DEBUG - Link is: /gp/customer-reviews/R123ICSCDM2MF3/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&ASIN=B07843GH8X
     2019-02-15 22:51:32,295 - DEBUG - Link is: /gp/customer-reviews/R3OTWE19SMPPVQ/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&ASIN=B07843GH8X
     2019-02-15 22:51:32,628 - DEBUG - Link is: /gp/customer-reviews/R1Z51ERFCD7D6P/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&ASIN=B07843GH8X
     2019-02-15 22:51:32,933 - DEBUG - Link is: /gp/customer-reviews/R24GYC4HRBGTM1/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&ASIN=B07843GH8X
     2019-02-15 22:51:33,302 - DEBUG - Link is: /gp/customer-reviews/R2P5ILE8KQF8PJ/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&ASIN=B07843GH8X
    

    如果您的调试函数抛出错误: 我不确定您的实际代码中的这个空间是否存在,或者它在此处以错误的帖子格式显示,但无论如何您应该从格式 arg '% (levelname)s' 中删除空格,以便调试按预期工作:

    替换这个:

    logging.basicConfig(level=logging.DEBUG, format=' %(asctime)s - %  (levelname)s - %(message)s')
    

    logging.basicConfig(level=logging.DEBUG, format=' %(asctime)s - %(levelname)s - %(message)s')
    

    希望这会对你有所帮助。

    【讨论】:

    • 是的,抱歉,空白只是错误的帖子格式
    • 好的,我看到你现在修复了帖子中的空白。你用过pycharm吗?当程序要求链接时,请在按 Enter 之前编写链接并添加空格。这将阻止 IDE 在新窗口中打开链接,而不是用返回键结束输入,并在输入函数后继续执行下一个代码。
    【解决方案2】:

    已经提到了空白问题。但是,您正在使用一个长且因此较慢的选择器,它也可能更脆弱。您可以使用性能更高的:

    linkElems = soup.select("a.review-title")
    

    甚至更快

    linkElems = soup.select(".review-title")
    

    类选择器是仅次于 id 的第二快的选择器方法。

    【讨论】:

    • 不幸的是,我已经尝试了所有这些都没有成功
    • 你能给我一个失败的网址吗?
    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2015-06-09
    • 2017-07-22
    • 2021-08-20
    • 2017-11-27
    相关资源
    最近更新 更多