满足某些条件时无法摆脱循环

Question

我在 python 中创建了一个脚本来获取来自 bing 的搜索结果的前 400 个链接。不确定是否总是至少有 400 个结果。在这种情况下，结果数约为 300。它的目标网页中有 10 个结果。但是，可以在遍历下一页时找到其余结果。问题是当那里没有更多的下一页链接时，网页会一遍又一遍地显示最后的结果。

搜索关键字是michael jackson，这是一个成熟的link

当没有更多新结果或结果小于 400 时，如何摆脱循环？`

我试过了：

import time
import requests
from bs4 import BeautifulSoup

link = "https://www.bing.com/search?"

params = {'q': 'michael jackson','first': ''}

def get_bing_results(url):
    q = 1
    while q<=400:
        params['first'] = q
        res = requests.get(url,params=params,headers={
            "User-Agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
            })
        soup = BeautifulSoup(res.text,"lxml")
        for link in soup.select("#b_results h2 > a"):
            print(link.get("href"))

        time.sleep(2)
        q+=10

if __name__ == '__main__':
    get_bing_results(link)

Answer 1

正如我在 cmets 中提到的，你不能这样做吗：

import time
import requests
from bs4 import BeautifulSoup

link = "https://www.bing.com/search?"

params = {'q': 'michael jackson','first': ''}

def get_bing_results(url):
    q = 1
    prev_soup = str()
    while q <= 400:
        params['first'] = q
        res = requests.get(url,params=params,headers={
            "User-Agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
            })
        soup = BeautifulSoup(res.text,"lxml")
        if str(soup) != prev_soup:
            for link in soup.select("#b_results h2 > a"):
                print(link.get("href"))
            prev_soup = str(soup)
        else:
            break
        time.sleep(2)
        q+=10

if __name__ == '__main__':
    get_bing_results(link)