遍历列表并仅从页面中提取最后一个链接的链接的尴尬问题 [BS4]

Question

我正在尝试抓取该网站，其中有 12 个带有 X 链接的页面 - 我只想提取所有链接，并将它们存储起来以备后用。

但是从页面中提取链接存在一个尴尬的问题。准确地说，我的输出仅包含每个页面的最后一个链接。

我知道这个描述可能听起来令人困惑，所以让我向您展示代码和图像：

import requests
from bs4 import BeautifulSoup
import pandas as pd
import csv
import time

#here I tried to make a loop for generating page's URLs, and store URLs in the list "issues" 

archive = '[redacted URL]'
issues =[]
#i am going for issues 163-175
for i in range(163,175): 
    url_of_issue = archive + '/' + str(i)
    issues.append(url_of_issue)

#now, I want to extract links from generated pages
#idea is simple - loop iterates over the list of URLs/pages and from each issue page get URLS of the listed papers, storing them in the list "paper_urls" 

paper_urls =[]

for url in issues: 
    response = requests.get(url)
    html = response.text
    soup = BeautifulSoup(html, "html.parser")
    for a in soup.select('.obj_article_summary  .title a'):
        ahrefTags=(a['href'])
    paper_urls.append(ahrefTags)
    print(paper_urls)
    time.sleep(5)

但问题是，我的输出看起来像 [redacted]。 而不是〜80个链接，我得到这个！我想知道发生了什么，看起来我的每个生成的 URL 的脚本（来自代码中名为“问题”的列表）只获得最后列出的链接？！如何解决？我不知道这里应该是什么问题。

Answer 1

在附加到paper_urls 时是否可能缺少缩进？

paper_urls =[]

for url in issues: 
    response = requests.get(url)
    html = response.text
    soup = BeautifulSoup(html, "html.parser")
    for a in soup.select('.obj_article_summary  .title a'):
        ahrefTags=(a['href'])
        paper_urls.append(ahrefTags)    # added missing indentation
    print(paper_urls)
    time.sleep(5)

将打印移出循环后，整个代码如下所示：

import requests
from bs4 import BeautifulSoup
import pandas as pd
import csv
import time

#here I tried to make a loop for generating page's URLs, and store URLs in the list "issues" 

archive = '[redacted URL]'
issues =[]
#i am going for issues 163-175
for i in range(163,175): 
    url_of_issue = archive + '/' + str(i)
    issues.append(url_of_issue)

#now, I want to extract links from generated pages
#idea is simple - loop iterates over the list of URLs/pages and from each issue page get URLS of the listed papers, storing them in the list "paper_urls" 

paper_urls =[]

for url in issues: 
    response = requests.get(url)
    html = response.text
    soup = BeautifulSoup(html, "html.parser")
    for a in soup.select('.obj_article_summary  .title a'):
        ahrefTags=(a['href'])
        paper_urls.append(ahrefTags)
        #print(ahrefTags)   #uncomment if you wish to print each and every link by itself
    #time.sleep(5)    #uncomment if you wish to add a delay between each request
print(paper_urls)