Beautifulsoup 解析 html 换行符

Question

我正在使用 BeautifulSoup 从文本文件中解析一些 HTML。文本被写入字典，如下所示：

websites = ["1"]

html_dict = {}

for website_id in websites:
    with codecs.open("{}/final_output/raw_html.txt".format(website_id), encoding="utf8") as out:   
        get_raw_html = out.read().splitlines()
        html_dict.update({website_id:get_raw_html})

我从 html_dict = {} 解析 HTML 以查找带有 <p> 标签的文本：

scraped = {}

for website_id in html_dict.keys():
    scraped[website_id] = []
    raw_html = html_dict[website_id]
    for i in raw_html:
        soup = BeautifulSoup(i, 'html.parser')
        scrape_selected_tags = soup.find_all('p')

这是html_dict 中的 HTML 的样子：

<p>Hey, this should be scraped
but this part gets ignored for some reason.</p>

问题是，BeautifulSoup 似乎正在考虑换行并忽略第二行。所以当我打印出scrape_selected_tags 时，输出是......

<p>Hey, this should be scraped</p>

当我期待全文时。

我怎样才能避免这种情况？我试过在html_dict 中拆分行，但它似乎不起作用。提前致谢。

Answer 1

通过在阅读 html 文档时调用 splitlines，您会破坏字符串列表中的标签。
相反，您应该阅读字符串中的所有 html。

websites = ["1"]
html_dict = {}

for website_id in websites:
    with codecs.open("{}/final_output/raw_html.txt".format(website_id), encoding="utf8") as out:   
        get_raw_html = out.read()
        html_dict.update({website_id:get_raw_html})

然后删除内部 for 循环，这样您就不会遍历该字符串。

scraped = {}

for website_id in html_dict.keys():
    scraped[website_id] = []
    raw_html = html_dict[website_id]
    soup = BeautifulSoup(raw_html, 'html.parser')
    scrape_selected_tags = soup.find_all('p')

---

BeautifulSoup 可以处理标签内的换行符，我举个例子：

html = '''<p>Hey, this should be scraped
but this part gets ignored for some reason.</p>'''

soup = BeautifulSoup(html, 'html.parser')
print(soup.find_all('p'))

[<p>Hey, this should be scraped\nbut this part gets ignored for some reason.</p>]

但如果你将一个标签拆分为多个BeautifulSoup 对象：

html = '''<p>Hey, this should be scraped
but this part gets ignored for some reason.</p>'''

for line in html.splitlines():
    soup = BeautifulSoup(line, 'html.parser')
    print(soup.find_all('p'))

[<p>Hey, this should be scraped</p>]
[]