如何抓取下一页（链接）

Question

我们现在通过此代码获得了网站www.theft-alerts.com 的第一页：

connection = urllib2.urlopen('http://www.theft-alerts.com')
soup = BeautifulSoup(connection.read().replace("<br>","\n"), "html.parser")

theftalerts = []
for sp in soup.select("table div.itemspacingmodified"):
    for wd in sp.select("div.itemindentmodified"):
        text = wd.text
        if not text.startswith("Images :"):
            print(text)

第一页的输出：

STOLEN : A LARGE TAYLORS OF LOUGHBOROUGH BELL
Stolen from Bromyard on 7 August 2014
Item : The bell has a diameter of 37 1/2" is approx 3' tall weighs just shy of half a ton and was made by Taylor's of Loughborough in 1902. It is stamped with the numbers 232 and 11.

The bell had come from Co-operative Wholesale Society's Crumpsall Biscuit Works in Manchester.
Any info to : PC 2361. Tel 0300 333 3000
Messages : Send a message
Crime Ref : 22EJ / 50213D-14

No of items stolen : 1

Location : UK > Hereford & Worcs
Category : Shop, Pub, Church, Telephone Boxes & Bygones
ID : 84377
User : 1 ; Antique/Reclamation/Salvage Trade ;  (Administrator)
Date Created : 11 Aug 2014 15:27:57
Date Modified : 11 Aug 2014 15:37:21;

网站上有更多页面（1 到 19）。我们只看到第 1 页。我们怎样才能得到其余的页面？

我们试过这个：

connection = urllib2.urlopen('http://www.theft-alerts.com', 'http://www.theft-alerts.com/index-2.html', 'http://www.theft-alerts.com/index-3.html', 'http://www.theft-alerts.com/index-4.html','http://www.theft-alerts.com/index-5.html', 'http://www.theft-alerts.com/index-6.html', 'http://www.theft-alerts.com/index-7.html')

但这不起作用。 输出：

"You can't pass both context and any of cafile, capath, and "
ValueError: You can't pass both context and any of cafile, capath, and cadefault

Answer 1

您可以通过访问带有resultnav 类的code 标记并遍历其中的a 标记来获取下一页的链接：

pages_nav = soup.find('code', class_='resultnav');
pages_links = pages_nav.find_all('a')
# Access `href` attribute after that

Answer 2

为什么不用索引号循环呢？

for i in range(1, 20):
    connection = urllib2.urlopen("http://www.theft-alerts.com/index-%i.html" % i0
    # process the file here

对于一个更通用的解决方案，它会一直持续到下一页不是有效链接：

i = 1
while True:
    conn = urllib2.urlopen("http://www.theft-alerts.com/index-%i.html" % i0
    if conn.getcode != 200:  # perhaps retry a couple of times
        break
    # process the file here
    i += 1

您的代码的问题是您试图将多个链接传递给urllib2.urlopen，但这不是它的工作原理。您需要传递每个链接，然后处理响应。

这是urlopen 的签名，应该可以解释您看到的错误：

def urlopen(url, data=None, timeout=socket.
            _GLOBAL_DEFAULT_TIMEOUT, 
            cafile=None, capath=None, cadefault=False, context=None)