BeautifulSoup 没有读取相同的 HTML 源代码

Question

我有一个网络抓取脚本已经工作了几个月，但今天却没有。调用时出现错误：

import bs4
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url = f'https://www.blocket.se/annonser/hela_sverige/fordon/bilar?cb=40&cbl1=6&cchb=1&ccsc=1&cg=1020&f=c&mye=2017&mys=2013&page=1&sort=date'
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()

page_soup = soup(page_html, "html.parser")
containers = page_soup.findAll("div", {"class":"styled__Wrapper-sc-1kpvi4z-0 itHtzm"})
print(len(containers))

这应该是 40 个元素，但它是：

0

现在下面的命令用来获取所有想要的容器，但是现在它什么也没找到。通过打印page_soup 变量，我发现该类已将名称更改为gSWafH 而不是itHtzm。

containers = page_soup.findAll("div", {"class":"styled__Wrapper-sc-1kpvi4z-0 gSWafH"})
print(len(containers))

反而给了想要的：

40

所有课程都发生了类似的变化，我首先认为网站发生了变化。但是，如果我自己阅读网站上的 HTML 代码，一切都没有改变。

为什么手动到站点找到的HTML代码和在浏览器中查看HTML代码和使用BS4读取有区别？

我知道我可以更改所有类名/搜索来修复脚本，但它是一个相当长的脚本，我更希望知道差异的原因。

Answer 1

您是否意识到您正在抓取的整个网站都在 JS 后面？话虽如此，bs4 不会看到任何东西，这就解释了你的结果。也就是说，你返回的HTML中没有这样的内容。

但是，您可以查询一个 API。

方法如下：

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:83.0) Gecko/20100101 Firefox/83.0",
    "Authorization": "Bearer 2381c6b987aea877abc6a73fe1cbc7d4a88a602c",
}

api = "https://api.blocket.se/search_bff/v1/content?cb=40&cbl1=6&cchb=1&ccsc=1&cg=1020&f=c&lim=40&mye=2017&mys=2013&page=0&sort=date"
response = requests.get(api, headers=headers).json()
print(len(response["data"]))

这应该打印40，这是报价的数量。

注意：我还不确定 Bearer 字符串的来源，但我会更深入地挖掘并返回这个答案。

顺便说一句，你想要的所有数据都在 JSON 中返回，例如你可以这样做：

import requests
from tabulate import tabulate

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:83.0) Gecko/20100101 Firefox/83.0",
    "Authorization": "Bearer 2381c6b987aea877abc6a73fe1cbc7d4a88a602c",
}

api = "https://api.blocket.se/search_bff/v1/content?cb=40&cbl1=6&cchb=1&ccsc=1&cg=1020&f=c&lim=40&mye=2017&mys=2013&page=0&sort=date"
response = requests.get(api, headers=headers).json()
print(len(response["data"]))

listing = [
    [
        i["subject"],
        i["price"]["value"],
        i["share_url"]
    ] for i in response["data"]
]

print(tabulate(listing, headers=["Name", "Price", "URL"]))

要得到这个：

Name                                                            Price  URL
------------------------------------------------------------  -------  --------------------------------------
Volkswagen Golf R 2.0 4Motion DSG Euro 6 300hk                 295000  https://www.blocket.se/vi/94476197.htm
Volkswagen Golf 1.6 TDI Style Eu6 110hk Nyservad               104900  https://www.blocket.se/vi/94476140.htm
Volkswagen Golf 1.4 150hk R-Line Nyservad 2Brukare 4.5/100km   173700  https://www.blocket.se/vi/94393769.htm
Volkswagen Golf GTD 2.0 TDI DSG Euro 6 184hk                   189000  https://www.blocket.se/vi/92041858.htm
Volkswagen Golf 1.4 TSI DSG R-LINE APPLE CARPLAY 150hk         179000  https://www.blocket.se/vi/94258178.htm
Volkswagen Golf R 2.0 4MOTION DSG MILLTEK PANORAMA GPS 300hk   289000  https://www.blocket.se/vi/94067452.htm
Volkswagen Golf 1.0 TSI BLUEMOTION DSG LÅGMIL 110hk            169000  https://www.blocket.se/vi/93849586.htm
Volkswagen Golf R 2.0 4M DSG Panorama Facelift Euro 6 310hk    329900  https://www.blocket.se/vi/94475139.htm
Volkswagen Golf TSi 140hk DSG /R-Line/Drag/Mok                 164900  https://www.blocket.se/vi/94473981.htm
Volkswagen Golf e-Golf 24.2 kWh /V-hjul/Navigation             199900  https://www.blocket.se/vi/91366004.htm
Volkswagen Golf Sportsvan 1.2 TSI DSG AUT Style EU6            159900  https://www.blocket.se/vi/93688078.htm
Volkswagen Golf 1.4 TSI Navi Drag Fullservad 140hk             122800  https://www.blocket.se/vi/94196494.htm
Volkswagen Golf GTE Hybrid 204hk B-kamera Drag Fullservad      199800  https://www.blocket.se/vi/94302825.htm
Volkswagen Golf 1.6 TDI 110HK Fjärrstyrd värmare               129000  https://www.blocket.se/vi/94433465.htm
Volkswagen Golf 1.2 TSI 110HK Aut Årsskatt 382kr               147500  https://www.blocket.se/vi/94057577.htm
Volkswagen Golf GTE 1.4 TSI DSG Sekventiell Euro 6 204hk       174900  https://www.blocket.se/vi/94469939.htm
Volkswagen Golf Alltrack 4M Eu6 184hk Premium D-värm B-kamer   164400  https://www.blocket.se/vi/94240149.htm
Volkswagen Golf 1.2 TSI BlueMotion 105hk Nyservad               84800  https://www.blocket.se/vi/94466632.htm
Volkswagen Golf 1.2 TSI 110 | Style | Komplett servicebok      114500  https://www.blocket.se/vi/92133274.htm
Volkswagen Golf 5-dr P-sensor 1.6 TDI 115hk                    144800  https://www.blocket.se/vi/93528128.htm
Volkswagen Golf 5-door R 2.0 4Motion DSG Skinn Pano Dynaudio   268900  https://www.blocket.se/vi/94134128.htm
Volkswagen Golf 1.6TDI M-värm Drag Eu6 110hk                   129800  https://www.blocket.se/vi/93438018.htm
Volkswagen Golf 1.2 TSI Fullservad 105hk                       104800  https://www.blocket.se/vi/94342484.htm
Volkswagen Golf 1.4 TSI Multifuel | Style | M-Värme | 5-dorr   116800  https://www.blocket.se/vi/93526752.htm
Volkswagen Golf GTI 2.0 TSI 230hk Performance Euro6            239900  https://www.blocket.se/vi/91536231.htm
Volkswagen Golf 140hk / Highline Plus                          114900  https://www.blocket.se/vi/89194496.htm
Volkswagen Golf 1.4 TSI / Style / 5-dörrar                      99000  https://www.blocket.se/vi/93718402.htm
Volkswagen Golf Sportsvan 1.2 TSI DSG 12 månaders garanti      169900  https://www.blocket.se/vi/93729451.htm
Volkswagen Golf Sportsvan TSI 110 MASTERS                      115000  https://www.blocket.se/vi/94455199.htm
Volkswagen Golf 1.2 TSI VÄLVÅRDAD STYLE 105hk                   89900  https://www.blocket.se/vi/94455152.htm
Volkswagen Golf 5-dörrar 1.6 TDI BlueMotion Design sport        72900  https://www.blocket.se/vi/94454862.htm
Volkswagen Golf 1.6 TDI Aut | Darklabel | P-värmare             99000  https://www.blocket.se/vi/94131096.htm
Volkswagen Golf 5-dr GTI Performance 2.0 TSI Eu 6 245hk        234500  https://www.blocket.se/vi/94453627.htm
Volkswagen Golf 5-d GTD 2.0 TDI Premium Kamera/Värmare/Drag    169000  https://www.blocket.se/vi/94453171.htm
Volkswagen Golf GTD 2.0 184hk D-Värmare Dynaudio Välservad     174900  https://www.blocket.se/vi/94452825.htm
Volkswagen Golf 1.6TDI 105hk Style P.sensor Välservad          127900  https://www.blocket.se/vi/94452680.htm
Volkswagen Golf TDI 150hk R-Line DSG / 1.99% Ränta             195000  https://www.blocket.se/vi/93752165.htm
Volkswagen Golf 5-dörrar 1.6 TDI Design, Style 105hk            89900  https://www.blocket.se/vi/94448403.htm
Volkswagen Golf 5-dörrar 1.6 TDI DSG Sekventiell Style 105hk   119900  https://www.blocket.se/vi/94448402.htm
Volkswagen Golf 5-door R 2.0 4Motion DSG 310hk                 309000  https://www.blocket.se/vi/94126802.htm

Answer 2

编辑：实际上我上面的那个人是对的，我没有研究它是否是通过 JS 加载的，因为我认为你可以通过正常的请求来抓取就好了。如果它是动态的，则此解决方案不起作用。

我刚刚检查了自己，似乎该站点确实将类名更改为随机变量。 只需要获取每个广告的数据吗？然后，您可以尝试获得第一堂课。从您的代码看起来没有改变：

containers = page_soup.findAll("div", {"class":"styled__Wrapper-sc-1kpvi4z-0"})

这获得与添加随机类相同数量的结果。通过检查开发控制台，顶部有一个滑块，可以解释原因。

我确实想知道你是如何在这两个课程中获得 40 个结果的，因为我得到了 47 个。