网页抓取 valueError BeautifulSoup

Question

所以我正在抓取一个[网站][1]，我想 根据这些 URL 检索网页并将每个网页转换为 beautifulsoup 对象

检索汽车制造年份、发动机、价格、经销商信息（如果有）以及访问详细汽车信息的 URL (href)。

当我运行代码时，我收到错误“ValueError：没有足够的值来解压（预期 4，得到 3）”，当我删除一个值而不是制造商、型号、年份和价格时，我将其更改为制造商、型号和price 并出现另一个错误“太多值无法解压（预期 3）”

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = "https://jammer.ie/used-cars?page={}&per-page=12"

all_data = []

for page in range(1, 3):  # <-- increase number of pages here
    soup = BeautifulSoup(requests.get(url.format(page)).text, "html.parser")

    for car in soup.select(".car"):
        info = car.select_one(".top-info").get_text(strip=True, separator="|")
        make, model, year, price = info.split("|")
        dealer_name = car.select_one(".dealer-name h6").get_text(
            strip=True, separator=" "
        )
        address = car.select_one(".address").get_text(strip=True)

        features = {}
        for feature in car.select(".car--features li"):
            k = feature.img["src"].split("/")[-1].split(".")[0]
            v = feature.span.text
            features[f"feature_{k}"] = v

        all_data.append(
            {
                "make": make,
                "model": model,
                "year": year,
                "price": price,
                "dealer_name": dealer_name,
                "address": address,
                "url": "https://jammer.ie"
                + car.select_one("a[href*=vehicle]")["href"],
                **features,
            }
        )

df = pd.DataFrame(all_data)
# prints sample data to screen:
print(df.tail().to_markdown(index=False))
# saves all data to CSV
df.to_csv('data.csv', index=False)

Answer 1

您可以检查汽车是否包含模型：

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = "https://jammer.ie/used-cars?page={}&per-page=12"

all_data = []

for page in range(1, 3):  # <-- increase number of pages here
    soup = BeautifulSoup(requests.get(url.format(page)).text, "html.parser")

    for car in soup.select(".car"):
        info = car.select_one(".top-info").get_text(strip=True, separator="|")
        info = info.split("|")
        if len(info) == 4:
            make, model, year, price = info
        else:
            make, year, price = info
            model = "N/A"
        dealer_name = car.select_one(".dealer-name h6").get_text(
            strip=True, separator=" "
        )
        address = car.select_one(".address").get_text(strip=True)

        features = {}
        for feature in car.select(".car--features li"):
            k = feature.img["src"].split("/")[-1].split(".")[0]
            v = feature.span.text
            features[f"feature_{k}"] = v

        all_data.append(
            {
                "make": make,
                "model": model,
                "year": year,
                "price": price,
                "dealer_name": dealer_name,
                "address": address,
                "url": "https://jammer.ie"
                + car.select_one("a[href*=vehicle]")["href"],
                **features,
            }
        )

df = pd.DataFrame(all_data)
# prints sample data to screen:
print(df.tail().to_markdown(index=False))
# saves all data to CSV
df.to_csv("data.csv", index=False)

印刷：

make	model	year	price	dealer_name	address	url	feature_speed	feature_engine	feature_transmission	feature_door-icon1	feature_petrol5	feature_hatchback	feature_owner	feature_paint
SEAT	Leon	2015	Price on application	McNamara Motors	Co. Cork	https://jammer.ie/vehicle/166591-seat-leon-2015	45000 miles	1.2 litres	Manual	5 doors	Petrol	Hatchback	2 previous owners	Grey
Toyota	Verso	2012	€8,250	Amcc	Co. Dublin	https://jammer.ie/vehicle/166590-toyota-verso-2012	98179 miles	1.5 litres	Automatic	4 doors	Petrol	MPV	nan	Purple
Mazda	Demio	2012	€7,950	Amcc	Co. Dublin	https://jammer.ie/vehicle/166589-mazda-demio-2012	82644 miles	1.3 litres	Automatic	4 doors	Petrol	Hatchback	nan	Red
Toyota	Corolla	2017	€14,950	Amcc	Co. Dublin	https://jammer.ie/vehicle/166588-toyota-corolla-2017	78916 miles	1.5 litres	Automatic	4 doors	nan	Estate	nan	Silver
Mazda	Demio	2013	€8,950	Amcc	Co. Dublin	https://jammer.ie/vehicle/166587-mazda-demio-2013	53439 miles	1.3 litres	Automatic	4 doors	Petrol	Hatchback	nan	Grey