使用 BeautifulSoup 在 Python 中获取新闻文章

Question

我正在尝试从 nbcnews.com 提取故事。我目前有以下代码：

import urllib2
from bs4 import BeautifulSoup

# The page that I'm getting stories from
url = 'http://www.nbcnews.com/'

data = urllib2.urlopen(url)

soup = BeautifulSoup(data, 'html.parser')

#This is the tag and class that chrome told me "top stories" are stored in
this = soup.find_all('div', attrs={"class": "col-sm-6 col-md-8 col-lg-9"})

#Get the a tags in the previous tag (this is the part that returns FAR too many links

link = [a for i in this for a in i.find_all('a')]

#Get the titles (This works)
title = [a.get_text() for i in link for a in i.find_all('h3')]

#The below strips all newlines and tabs from the title name
newtitle = []

for i in t:
    s = ' '.join(i.split())
    if s in newtitle:
        pass

    else:
        newtitle.append(s)

print len(link)
print len(title)

当我运行脚本时，“标题”列表（大部分）是正确的，但网站上的标题名称略有不同（如果标题名称接近相同，则没有问题）

我的问题是“链接”列表似乎包含来自各地的链接？有人可以帮我弄这个吗？

或者如果可能的话，是否有类似的 API 可用？如果可以避免的话，我真的不想重新发明*来获取新闻文章。

编辑：更改了变量名中的拼写错误

Answer 1

查看有问题的网页，看起来所有新闻都在 h3 标签中，类为 item-heading。您可以使用 BeautifulSoup 选择所有故事标题，然后使用 BeautifulSoup 的 .parent 方法在 HTML 树中向上并访问它们包含在其中的 a href：

In [54]: [i.parent.attrs["href"] for i in soup.select('a > h3.item-heading')]
Out[55]:
[{'href': '/news/us-news/civil-rights-groups-fight-trump-s-refugee-ban-uncertainty-continues-n713811'},
 {'href': '/news/us-news/protests-erupt-nationwide-second-day-over-trump-s-travel-ban-n713771'},
 {'href': '/politics/politics-news/some-republicans-criticize-trump-s-immigration-order-n713826'},
...  # trimmed for readability
]

我使用了列表推导，但您可以拆分为复合步骤：

# select all `h3` tags with the matching class that are contained within an `a` link.
# This excludes any random links elsewhere on the page.
story_headers = soup.select('a > h3.item-heading')

# Iterate through all the matching `h3` items and access their parent `a` tag.
# Then, within the parent you have access to the `href` attribute.
list_of_links = [i.parent.attrs for i in story_headers]

# Finally, extract the links into a tidy list
links = [i["href"] for i in list_of_links]

获得链接列表后，您可以遍历它以检查第一个字符是否为/，以仅匹配本地链接而不匹配外部链接。