要求用户输入内容并使用 Beautiful Soup 解析网站

Question

我应该使用 Beautiful Soup 4 从我学校的网站上获取课程信息作为练习。过去几天我一直在这，我的代码仍然无法正常工作。

我要求用户的第一件事是导入课程目录缩写。例如，ICS 缩写为 Information for Computer Science。 Beautiful Soup 4 应该列出所有的课程以及有多少学生注册。

虽然我能够让输入部分工作，但我仍然有错误或程序停止。

问题：Beautiful Soup 有没有办法接受用户输入，这样当用户输入 ICS 时，输出将是与 ICS 相关的所有课程的列表？

这是代码和我的尝试：

from bs4 import BeautifulSoup
import requests
import re

#get input for course
course = input('Enter the course:')
#Here is the page link
BASE_AVAILABILITY_URL = f"https://www.sis.hawaii.edu/uhdad/avail.classes?i=MAN&t=202010&s={course}"


#get request and response
page_response = requests.get(BASE_AVAILABILITY_URL)
#getting Beautiful Soup to gather the html content
page_content = BeautifulSoup(page_response.content, 'html.parser')
#getting course information
main = page_content.find_all(class_='parent clearfix')
main_p = "".join(str (x) for x in main)
#get the course anchor tags
main_q = BeautifulSoup(main_p, "html.parser")
courses = main.find('a', href = True)
#get each course name
#empty dictionary for course list
courses_list = []
for a in courses:
    courses_list.append(a.text)
    search = input('Enter the course title:')
for course in courses_list:
    if re.search(search, course, re.IGNORECASE):
        print(course)

这是 Juypter Notebook 中提供的原始代码

import requests, bs4

BASE_AVAILABILITY_URL = f"https://www.sis.hawaii.edu/uhdad/avail.classes?i=MAN&t=202010&s={course}"
#get input for course
course = input('Enter the course:')



def scrape_availability(text):
    soup = bs4.BeautifulSoup(text)
    r = requests.get(str(BASE_AVAILABILITY_URL)  + str(course))
    rows = soup.select('.listOfClasses tr')

    for row in rows[1:]:
        columns = row.select('td')
        class_name = columns[2].contents[0]
        if len(class_name) > 1 and class_name != b'\xa0':
            print(class_name)
            print(columns[4].contents[0])
            print(columns[7].contents[0])
            print(columns[8].contents[0])

奇怪的是，如果用户保存 html 文件，将其上传到 Juypter Notebook，然后打开要阅读的文件，就会显示课程。但是，对于这个任务，用户不能保存文件，它必须是一个直接的输入才能得到输出。

Answer 1

您的代码的问题是page_content.find_all(class_='parent clearfix') retuns 和空列表[]。所以这是你需要改变的第一件事。查看 html，您会想要寻找 <table>、<tr>、<td>、标签

处理原始代码提供的内容，您只需要更改一些内容以符合逻辑：

我会指出我改变了什么：

import requests, bs4

BASE_AVAILABILITY_URL = f"https://www.sis.hawaii.edu/uhdad/avail.classes?i=MAN&t=202010&s={course}"
#get input for course
course = input('Enter the course:')



def scrape_availability(text):
    soup = bs4.BeautifulSoup(text)   #<-- need to get the html text before creating a bs4 object. So I move the request (line below) before this, and also adjusted the parameter for this function.
                                     # the rest of the code is fine
    r = requests.get(str(BASE_AVAILABILITY_URL)  + str(course))
    rows = soup.select('.listOfClasses tr')

    for row in rows[1:]:
        columns = row.select('td')
        class_name = columns[2].contents[0]
        if len(class_name) > 1 and class_name != b'\xa0':
            print(class_name)
            print(columns[4].contents[0])
            print(columns[7].contents[0])
            print(columns[8].contents[0])

这会给你：

import requests, bs4

BASE_AVAILABILITY_URL = "https://www.sis.hawaii.edu/uhdad/avail.classes?i=MAN&t=202010&s="
#get input for course
course = input('Enter the course:')

url = BASE_AVAILABILITY_URL  + course

def scrape_availability(url):

    r = requests.get(url)
    soup = bs4.BeautifulSoup(r.text, 'html.parser')
    rows = soup.select('.listOfClasses tr')

    for row in rows[1:]:
        columns = row.select('td')
        class_name = columns[2].contents[0]
        if len(class_name) > 1 and class_name != b'\xa0':
            print(class_name)
            print(columns[4].contents[0])
            print(columns[7].contents[0])
            print(columns[8].contents[0])



scrape_availability(url)