循环遍历标签 - BeautifulSoup答案

【问题标题】：Looping through Tag - BeautifulSoup循环遍历标签 - BeautifulSoup
【发布时间】：2018-10-22 22:03:46
【问题描述】：

我试图从我的脚本中的网页中获取主题标题，但它只是拉出页面上的最后一个主题并将其添加到我的 JSON 文件中的每条记录中。如何让它遍历每个 h2 标签以将正确的主题添加到每个记录？

from urllib.request import urlopen
from bs4 import BeautifulSoup as soup
import json

uci_urls = ['http://ocw.uci.edu/courses']

data =[]
#opening up connection and grabbing page
for uci_url in uci_urls:
    uClient = urlopen(uci_url)
    page_html = uClient.read()
    uClient.close()

    #html parsing
    page_soup = soup(page_html, "html.parser")

    #grabs info for each textbook
    containers = page_soup.findAll("div",{"class":"panel-heading"})
    subjects = page_soup.findAll("h2")

    for container in containers:
        item = {}
        item['type'] = "Course"
        item['title'] = container.h3.text
        item['author'] = "University of California Irvine"
        item['link'] = "http://ocw.uci.edu/courses" + container.a["href"]
        item['source'] = "UCI Open"
        for subject in subjects:
           item['subject'] = subject.text
        item['base_url'] = "http://ocw.uci.edu/"
        data.append(item) # add the item to the list

    with open("./json/uci.json", "w") as writeJSON:
       json.dump(data, writeJSON, ensure_ascii=False)

【问题讨论】：

标签： python json beautifulsoup

【解决方案1】：

一次从页面中获取所有标题是无济于事的。您必须让每个父 div 包含所需的标题和课程面板。

#html parsing
page_soup = soup(page_html, "html.parser")

for tab_pane in page_soup.select('.col-sm-9.col-md-9.tab-content > .tab-pane'):
    subject = tab_pane.h2.text
    for container in tab_pane.findAll("div",{"class":"panel-heading"}):
        item = {}
        item['type'] = "Course"
        item['title'] = container.h3.text
        item['author'] = "University of California Irvine"
        item['link'] = "http://ocw.uci.edu/courses" + container.a["href"]
        item['source'] = "UCI Open"
        item['subject'] = subject
        item['base_url'] = "http://ocw.uci.edu/"
        data.append(item) # add the item to the list

【讨论】：