从维基百科文章中获取所有标题和纯文本

Question

在 Python 中，我将如何从 wikipedia 文章中获取所有标题和平面文本，例如：https://en.wikipedia.org/wiki/Amadeus_(film)。我目前的代码是这样的：

    from bs4 import BeautifulSoup


# ---- Definitions ----#
#Amount of documents
amount_of_documents = 1

#Directory of raw HTML documents
directory_of_raw_documents = "raw_documents/"

#Directory of parsed documents
directory_of_parsed_documents = "parsed_documents/"

# ---- Code ----#


def open_document():
    for i in range (1, 1+1):
        with open(directory_of_raw_documents + str(i), "r") as document:
            html = document.read()
            soup = BeautifulSoup(html, "html.parser")
            body = soup.find('div', id='bodyContent')
            for elements in body.find_all('p'):
                print(elements.text)

open_document()

我正在加载一个下载的 HTML 文件，然后使用 BeautifulSoup 抓取<p> 标签之间的所有内容。我的目标是获取这篇文章的所有标题和纯文本内容。我该怎么做呢？

在上面发布的示例中，我想要的输出是包含：

1. 所有标题（Amadeus（电影）、情节、演员、接待处等）
2. 本页内的所有文字（<p> 标签之间）
3. 忽略引用

Answer 1

您可能对使用专门的维基百科页面解析器感兴趣，例如 wikipedia package。这样您就可以轻松获取内容：

In [1]: import wikipedia

In [2]: page = wikipedia.page("Amadeus (film)")

In [3]: page.summary
Out[3]: u"Amadeus is a 1984 American period drama film directed by Milo\u0161 Forman, written by Peter Shaffer, and adapted from Shaffer's stage play Amadeus (1979). The story, set in Vienna, Austria, during the latter half of the 18th century, is a fictionalized biography of Wolfgang Amadeus Mozart. Mozart's music is heard extensively in the soundtrack of the movie. Its central thesis is that Antonio Salieri, an Italian contemporary of Mozart is so driven by jealousy of the latter and his success as a composer that he plans to kill him and to pass off a Requiem, which he secretly commissioned from Mozart as his own, to be premiered at Mozart's funeral. Historically, the Requiem which was never finished was commissioned by Count von Walsegg and Salieri, far from being jealous of Mozart, was on good terms with him and even tutored his son after Mozart's death.\nThe film was nominated for 53 awards and received 40, which included eight Academy Awards (including Best Picture), four BAFTA Awards, four Golden Globes, and a Directors Guild of America (DGA) award. As of 2016, it is the most recent film to have more than one nomination in the Academy Award for Best Actor category. In 1998, the American Film Institute ranked Amadeus 53rd on its 100 Years... 100 Movies list."

In [4]: page.content
Out[4]: u'Amadeus is a 1984 American period drama film directed by Milo\u0161 Forman, written by Peter Shaffer, and adapted from Shaffer\'s s
...
Amadeus Filming locations at Movieloci.com'

---

关于获取标题，这里是通过BeautifulSoup获取它们的示例代码：

In [1]: import requests

In [2]: from bs4 import BeautifulSoup

In [3]: url = "https://en.wikipedia.org/wiki/Amadeus_(film)"

In [4]: response = requests.get(url)

In [5]: soup = BeautifulSoup(response.content, "html.parser")

In [6]: [item.get_text() for item in soup.select("h2 .mw-headline")]
Out[6]: 
[u'Plot',
 u'Cast',
 u'Production',
 u'Reception',
 u'Alternative versions',
 u'Music',
 u'Awards and nominations',
 u'References',
 u'External links']

h2 .mw-headline 是一个 CSS selector，它将匹配具有 mw-headline 类在 h2 父元素下的元素。