美丽的汤不适用于 requests.get

Question

所以我是一个 python 初学者，试图抓取这个网站http://www.edwaittimes.ca/WaitTimes.aspx 这给了医院等待时间。目前我正在尝试打印所有医院的名称。

如果 .html 文件位于我正在使用的 python 文件所在的文件夹中，我的代码就可以工作

from bs4 import BeautifulSoup
import requests


def print_hospitals():
    with open('website.html','r') as html_file:
        content = html_file.read()
        soup = BeautifulSoup(content, 'lxml')
        hospital_table = soup.find_all('div',class_="Row")
        for hospital in hospital_table:
            if hospital.a is not None:
                print(hospital.a.text)

但是当我将 requests.get 与 URL 一起使用时。该代码不打印任何内容。也没有错误消息。

from bs4 import BeautifulSoup
import requests

def print_hospitals_request():
    html_text = requests.get('http://www.edwaittimes.ca/WaitTimes.aspx').text
    soup = BeautifulSoup(html_text, 'lxml')
    hospital_table = soup.find_all('div',class_="Row")
    for hospital in hospital_table:
        if hospital.a is not None:
            print(hospital.a.text)

谁能帮我解决这个问题

Answer 1

该页面使用 Ajax 从外部 URL 加载数据。所以beautifulsoup什么也没看到。要加载数据，您可以使用下一个示例：

import requests
from bs4 import BeautifulSoup


hospitals_csv = "http://www.edwaittimes.ca/Shared/Images/sites2.csv"

data = [
    l.split("|")[:-1]
    for l in requests.get(hospitals_csv).text.splitlines()[:-1]
]

all_data = ""
for hospital, city in data:
    url = (
        "http://www.edwaittimes.ca/Shared/Images/"
        + hospital
        + (".html" if city == "Vancouver" else "_gp.html")
    )
    print(f"Getting {url}")
    all_data += requests.get(url).text

soup = BeautifulSoup(all_data, "html.parser")
for row in soup.select(".Row"):
    print(row.get_text(strip=True, separator=" "))

印刷：

Lions Gate Hospital Patients of all ages seen 02:28 05:06
North Van Urgent & Primary Care Centre Patients of all ages seen UPCC is for mild to moderate illness 01:38 04:15
Squamish General Hospital Patients of all ages seen 01:39 02:16
Whistler Health Care Centre Patients of all ages seen 00:43 01:52
Pemberton Health Centre Patients of all ages seen No patients seen in the last two hours. 02:05
Sechelt Hospital Patients of all ages seen 02:08 04:52
Richmond Hospital Patients of all ages seen 02:36 05:16
Richmond Urgent and Primary Care Centre Patients of all ages seen (lab offsite) UPCC is for mild to moderate illness 03:46 03:28
Vancouver General Hospital Patients of ages 17 and older seen 02:18 05:40
St. Paul's Hospital Patients of all ages seen 00:34 04:26
Mount Saint Joseph Hospital Patients of all ages seen 01:01 02:35
UBC Hospital (UBCH) Patients of all ages seen UBCH is for mild to moderate illness 01:22 01:22
City Centre Urgent & Primary Care Centre Patients of all ages seen UPCC is for mild to moderate illness 01:49 02:30
REACH Urgent and Primary Care Centre Patients of all ages seen (lab & x-ray offsite) UPCC is for mild to moderate illness Currently open, call (604) 216-3138 for wait time
Northeast Urgent and Primary Care Centre Patients of all ages seen (lab & x-ray offsite) UPCC is for mild to moderate illness 02:50 02:50
Southeast Urgent and Primary Care Centre Patients of all ages seen (lab & x-ray offsite) UPCC is for mild to moderate illness 02:12 01:52
BC Children's Hospital Patients seen up to age 16 02:23 04:39

Answer 2

您要查找的类似乎不存在于您正在抓取的网页上。我将 class_="Row" 替换为 class_="grid_8"，这是网页上存在的一个类，并且有效：

from bs4 import BeautifulSoup
import requests


def print_hospitals_request():
    html_text = requests.get('http://www.edwaittimes.ca/WaitTimes.aspx').text
    soup = BeautifulSoup(html_text, 'lxml')
    hospital_table = soup.find_all('div', class_="grid_8")
    for hospital in hospital_table:
        if hospital.a is not None:
            print(hospital.a.text)


print_hospitals_request()

Answer 3

Beautiful Soup 和请求工作正常。你在理论上所做的工作。事情是这样的，您正在阅读的 html 是网站本身发出另一个请求然后根据该请求填充表格的结果。如果您进入并使用浏览器上的开发人员工具，您将看到一个具有特定操作的表单元素。我的猜测是获取请求填充用户看到的初始 html，然后是表单请求和一些 javascript 从服务器获取数据。

没有错误，因为这是 get 请求的结果。我不确定调用该表单的发布请求会做什么，而且我不确定该网站的使用条款或条件。

假设您确实有使用该 API 的权限，这不仅仅是无聊的好奇心。您可以选择两条路线之一。您可以尝试使用 get 而不是 post 来模拟页面发出的请求。另一种是使用 selenium（通过 python 绑定或其他方法）打开浏览器，call a wait till some element is present or a timeout occurs，然后使用 selenium 代替 bs4 来抓取页面。

如果这是为了练习，我在维基百科上使用了 bs4，这是一个很好的开放内容来源，其中包含大量表格并发送了所有原始 html。