Python Web 抓取 - 在 www.truecar.com 中查找特殊品牌的模型

Question

我试图用 bs4 库在 python 中抓取特殊数据。

https://www.truecar.com/used-cars-for-sale/listings/suzuki/location-chicago-il/

我想抓取一个特殊品牌的型号名称（例如 Suzuki）。 当我们在网页中打开检查元素并选择“模型”元素时。最后，我们得到了以下几行：

<option selected="" value="">All</option>

<option value="aerio">Aerio</option>

<option value="equator">Equator</option>

<option value="esteem">Esteem</option>

<option value="forenza">Forenza</option>

<option value="grand-vitara">Grand Vitara</option>

<option value="kizashi">Kizashi</option>

<option value="reno">Reno</option>

<option value="sx4">SX4</option>

等
所以根据我所学，我写了这个脚本：

import requests
from bs4 import BeautifulSoup


respose = requests.get('https://www.truecar.com/used-cars-for-sale/listings/suzuki/location-chicago-il/')
soup = BeautifulSoup(respose.text , 'html.parser')

models = soup.find_all('select',attrs={"class":"form-control-select-md form-select-element",
                                       "data-test":"searchFiltersModel"})

但在模型变量中，我们只有以下文本：

<select aria-label="Change vehicle model" class="form-control-select-md form-select-element" data-test="searchFiltersModel" data-qa="ModelFilter">
<option selected="" value="">All</option>
</select>

为什么会发生这种情况以及解决方案是什么？

Answer 1

如上所述，汽车模型是在您选择品牌后呈现的。仅访问静态 url 不会呈现它。这里更好的选择是使用 api。

注意，我在这里使用了一个包 choice 来获取输入有效负载所需的 slug：

import requests

#pip install choice  <--- TO INSTALL CHOICE
import choice

url = 'https://www.truecar.com/abp/api/vehicles/used/listings/aggregations_challenger?new_or_used=u'
jsonData = requests.get(url).json()

makes = jsonData['makes']
makes_dict = {}
for each in makes:
    makes_dict[each['name']] = each['slug'] 

make_key = choice.Menu(makes_dict.keys()).ask()
make_slug = makes_dict[make_key]

url = 'https://www.truecar.com/abp/api/vehicles/used/listings/aggregations_challenger'

payload = {'make_slug': make_slug}

jsonData = requests.get(url, params=payload).json()

for each in jsonData['makes']:
    if each['name'] == make_key:
        models = [x['name'] for x in each['models']]
        break

print(models)

输出：

 0: Saturn
 1: Scion
 2: smart
 3: Subaru
 4: Suzuki
 5: Tesla
 6: Toyota
 7: Volkswagen
 8: Volvo

Enter number or name; return for next page

? 4

['Aerio', 'Equator', 'Esteem', 'Forenza', 'Grand Vitara', 'Kizashi', 'Reno', 'SX4', 'Verona', 'Vitara', 'X-90', 'XL7']

如果您想一次性获得所有这些，只需遍历 make_slugs：

import requests

url = 'https://www.truecar.com/abp/api/vehicles/used/listings/aggregations_challenger'
jsonData = requests.get(url).json()

makes = jsonData['makes']
makes_dict = {}
for each in makes:
    makes_dict[each['name']] = each['slug'] 


for k,v in makes_dict.items():
    payload = {'make_slug': v}
    jsonData = requests.get(url, params=payload).json()

    for each in jsonData['makes']:
        if each['name'] == k:
            models = [x['name'] for x in each['models']]
            break
    
    print(k,': ',models)

输出：

Acura :  ['CL', 'ILX', 'Integra', 'MDX', 'NSX', 'RDX', 'RL', 'RLX', 'RSX', 'TL', 'TLX', 'TSX', 'ZDX']
Alfa Romeo :  ['4C', 'Giulia', 'Stelvio']
AM General :  ['Hummer']
Aston Martin :  ['DB11', 'DB7 Vantage', 'DB9', 'DBS', 'DBX', 'Rapide', 'Rapide S', 'V8 Vantage', 'Vanquish', 'Vanquish S', 'Vantage', 'Virage']
Audi :  ['A3', 'A4', 'A4 allroad', 'A5', 'A6', 'A6 allroad', 'A7', 'A8', 'allroad', 'e-tron', 'e-tron GT', 'Q3', 'Q5', 'Q7', 'Q8', 'R8', 'RS 3', 'RS 4', 'RS 5', 'RS 6 Avant', 'RS 7', 'RS Q8', 'RS6', 'S3', 'S4', 'S5', 'S6', 'S7', 'S8', 'SQ5', 'SQ7', 'SQ8', 'TT', 'TT RS', 'TTS']
Bentley :  ['Arnage', 'Azure', 'Bentayga', 'Continental', 'Continental GT', 'Continental Supersports', 'Flying Spur', 'Mulsanne']
BMW :  ['1 Series', '1 Series M', '2 Series', '3 Series', '4 Series', '5 Series', '6 Series', '7 Series', '8 Series', 'i3', 'i8', 'M2', 'M3', 'M4', 'M5', 'M6', 'M8', 'X1', 'X2', 'X3', 'X3 M', 'X4', 'X4 M', 'X5', 'X5 M', 'X6', 'X6 M', 'X7', 'Z3', 'Z4', 'Z8']
Buick :  ['Cascada', 'Century', 'Enclave', 'Encore', 'Encore GX', 'Envision', 'LaCrosse', 'LeSabre', 'Lucerne', 'Park Avenue', 'Rainier', 'Regal', 'Regal Sportback', 'Regal TourX', 'Rendezvous', 'Terraza', 'Verano']
Cadillac :  ['ATS', 'ATS-V', 'Catera', 'CT4', 'CT5', 'CT6', 'CT6-V', 'CTS', 'CTS-V', 'DeVille', 'DeVille Professional', 'Deville Professional', 'DTS', 'DTS Professional', 'Eldorado', 'ELR', 'Escalade', 'Escalade EXT', 'Seville', 'SRX', 'STS', 'STS-V', 'XLR', 'XLR-V', 'XT4', 'XT5', 'XT6', 'XTS']
...