如何使用 Python 从网站中读取所有纯文本？答案

【问题标题】：How can I read all the plain text from a website with Python?如何使用 Python 从网站中读取所有纯文本？
【发布时间】：2022-01-15 20:38:30
【问题描述】：

我正在尝试让我的代码打印网站上的所有纯文本。这是我的代码：

import requests
import json
response = requests.get("https://example.com")
json_data = json.loads(response.text)
print(str(json_data))

例如：如果我输入https://example.com，我想让程序写

Example Domain

This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.

More information...

但我收到此错误消息：

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    json_data = json.loads(response.text)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

我该如何解决这个问题？

【问题讨论】：

你为什么打电话给json.loads()？网站响应通常不是 json 格式。

标签： python json web python-requests

【解决方案1】：

from urllib.request import urlopen
from bs4 import BeautifulSoup

url = "https://example.com"
html = urlopen(url).read()
soup = BeautifulSoup(html, features="html.parser")
# delete unwanted elements
for script in soup(["script", "style"]):
    script.extract()  

# get actual text
text = soup.get_text()


lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
text = '\n'.join(chunk for chunk in chunks if chunk)

print(text)

别忘了先pip install beautifulsoup4

【讨论】：