在 For 循环内使用 Try/Except 更改 BeautifulSoup URL

Question

我正在抓取一个经常会收到AttributeError 的网站。发生这种情况时，我需要通过在循环遍历的 ID 列表中添加几个前导零来重试 URL。

import requests
from bs4 import BeautifulSoup as bs

ids = ['23135106', '37833100', '57636Q104']

base_url = "https://quotes.fidelity.com/mmnet/SymLookup.phtml?reqforlookup=REQUESTFORLOOKUP&productid=mmnet&isLoggedIn=mmnet&rows=50&for=stock&by=cusip&criteria="

# Create empty list to store scraped symbols
symbols = []
for id in ids:
    url = base_url + id + "&submit=Search"
    r = requests.get(url)
    soup = bs(r.content, 'lxml')
    try:
        symbol = soup.select_one('[href*=SID_VALUE_ID]').text
        print(symbol)
    except AttributeError:
        print("N/A")
    else:
        symbols.append(symbol)

因此，我不想在异常上打印N/A，而是想通过添加前导零来重试ID（例如，23135106 变为023135106，这是有效的），然后如果失败，重试通过添加两个前导零等。在某些情况下，ID 将失败，尽管添加了多少前导零，此时可以返回 N/A。

我怎样才能做到这一点？

Answer 1

只需使用另一个 for 循环：

import requests
from bs4 import BeautifulSoup

IDS = ['23135106', '37833100', '57636Q104']
BASE_URL = "https://quotes.fidelity.com/mmnet/SymLookup.phtml?reqforlookup=REQUESTFORLOOKUP&productid=mmnet&isLoggedIn=mmnet&rows=50&for=stock&by=cusip&criteria={}&submit=Search"

def read_symbol(id):
    r = requests.get(BASE_URL.format(id))
    soup = BeautifulSoup(r.content, 'lxml')
    symbol = soup.select_one('[href*=SID_VALUE_ID]')
    return symbol.text if symbol is not None else None

symbols = []
for id in IDS:
    for zeros in range(11):
        symbol = read_symbol(zeros * "0" + id)
        if symbol is not None:
            print(symbol)
            symbols.append(symbol)
            break