从请求 Python 中获得不需要的响应答案

【问题标题】：Getting Unwanted response From requsets Python从请求 Python 中获得不需要的响应
【发布时间】：2020-05-13 22:28:55
【问题描述】：

我正在尝试得到响应 https://us.vestiairecollective.com/members/profile-2241096.shtml#currentpgn=2 但是当使用 requests get 方法访问时，我得到了响应 https://us.vestiairecollective.com/members/profile-2241096.shtml#currentpgn=0 当前pgn的每个参数号都面临这个问题

from requests import get
from bs4 import BeautifulSoup
headers = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'
        }
resp = get("https://us.vestiairecollective.com/members/profile-2241096.shtml#currentpgn=2", headers=headers)
soup = BeautifulSoup(resp.text, 'lxml')
divs = soup.find('div', class_='catalog-list medium').find_all("div", recursive=False)
for div in divs:
    print(div.a['href'])```

【问题讨论】：

你能显示一些输出吗？我得到了预期的输出。

标签： python python-3.x web-scraping beautifulsoup python-requests

【解决方案1】：

数据通过 Ajax 调用加载。但是您可以使用 requests 模块复制 ajax 调用。例如（这会打印产品品牌、名称和价格的前 10 页）：

import re
import requests
from bs4 import BeautifulSoup

url = 'https://us.vestiairecollective.com/members/profile-2241096.shtml#currentpgn=2'

ajax_url = 'https://us.vestiairecollective.com/profil.shtml'

id_profil = re.search(r'-(\d+)\.', url).group(1)
headers = {'User-Agent':'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0'}
data = {'ajx':'1', 'limit':'60', 'step':'2', 'ajax_var':'sell', 'id_profil':id_profil, 'filterSold':'1'}

for page in range(0, 10):
    print('Processing page {}...'.format(page))
    print('-' * 80)
    data['step'] = page
    soup = BeautifulSoup(requests.post(ajax_url, data=data, headers=headers).json()['result'], 'html.parser' )


    for brand, name, price in zip(soup.select('.productItem .brand'),
                                  soup.select('.productItem .name'),
                                  soup.select('.productItem .price')):
        print(brand.text, name.text, price.text)

打印：

Processing page 0...
--------------------------------------------------------------------------------
LOUIS VUITTON Patent leather flip flops 226 €
JEAN PAUL GAULTIER Jacket 150 €
MIU MIU Leather hair accessory 299 €
CHANEL Belt 696 €
ETRO Hair accessory 149 €
DOLCE & GABBANA Mink bag charm 285 €
CHANEL Leather heels 636 €
CHANEL Silk shirt 490 €
VALENTINO GARAVANI Patent leather wallet 199 €

... and so on.

【讨论】：