亚马逊抓取分页随机失败答案

【问题标题】：Amazon scrape pagination random fail亚马逊抓取分页随机失败
【发布时间】：2021-09-02 17:19:13
【问题描述】：

import requests
from bs4 import BeautifulSoup
def getdata(url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_5_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"}
    r = s.get(url, headers=headers).text
    soup = BeautifulSoup(r, 'html.parser')
    return soup
def getnextpage(soup):
    try:
        div = soup.find('div',{'class': 'a-section a-spacing-large a-spacing-top-large a-text-center s-pagination-container'})
        url_a = div.find('a', {'class': 's-pagination-item s-pagination-next s-pagination-button s-pagination-separator'})['href']
        url = 'https://www.amazon.in' + url_a
        return url
    except:
        print("All pages scraped")
s = requests.Session()
url = f'https://www.amazon.in/s?k=apple&i=electronics&ref=nb_sb_noss_2'
while True:
    soup = getdata(url)
    #getdeals(soup)
    url = getnextpage(soup)
    print(url)

我正在刮亚马逊。但是，有时会找到下一页的a 元素，有时却找不到。我无法理解这其中的原因。如果找到它，它会继续运行所有页面，否则它会在第一页本身失败。我该如何纠正这个问题？

【问题讨论】：

标签： python web-scraping beautifulsoup python-requests

【解决方案1】：

亚马逊似乎返回了 2 个版本的 HTML 页面。我已经修改了getnextpage() 函数来尝试不同的元素来获取下一页 URL：

import requests
from bs4 import BeautifulSoup


def getdata(url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_5_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"
    }
    r = s.get(url, headers=headers).text
    soup = BeautifulSoup(r, "html.parser")
    return soup


def getnextpage(soup):
    next_ = soup.select_one("li.a-last a")
    if not next_:
        next_ = soup.select_one(".s-pagination-next")
        if not next_:
            return

    return "https://www.amazon.in" + next_["href"]


s = requests.Session()
url = f"https://www.amazon.in/s?k=apple&i=electronics&ref=nb_sb_noss_2"
while True:
    soup = getdata(url)
    # getdeals(soup)
    url = getnextpage(soup)
    if not url:
        break
    print(url)

打印：

https://www.amazon.in/s?k=apple&i=electronics&page=2&qid=1630604251&ref=sr_pg_1
https://www.amazon.in/s?k=apple&i=electronics&page=3&qid=1630604252&ref=sr_pg_2
https://www.amazon.in/s?k=apple&i=electronics&page=4&qid=1630604253&ref=sr_pg_3

...

【讨论】：