使用 BeautifulSoup 解析网站

Question

对于我的学校项目，我正在尝试从该站点 (http://www.boxofficemojo.com/monthly/?view=releasedate&chart=&month=1&yr=2006) 抓取一些数据。我浏览了beautifulsoup 的文档，但我现在遇到了问题。 这是目前的设置。

import urllib
import re
from bs4 import BeautifulSoup

html_source = urllib.urlopen('http://www.boxofficemojo.com/monthly/?
view=releasedate&chart=&month=1&yr=2006').read()

soup = BeautifulSoup(html_source, 'lxml')

我首先尝试获取标题，但是当我查看 html 代码时，我注意到标签既不包含 id 也不包含类。所以一开始，我尝试了soup.find_all(href=re.compile("movies"))，因为标签看起来像<td><b><font size="2"><a href="/movies/?id=bigmommashouse2.htm">Big Momma's House 2</a></font></b></td>，并且对于标题部分，href 总是以“/movies”开头。但是正如我所指出的，这不仅给了我标题，而且还给了我页面顶部不必要的价值，因为标签看起来几乎相同 <a href="/movies/?id=beautyandthebeast2017.htm">#1 Movie: 'Beauty and the Beast'</a>

然后我尝试了soup.select("td b font a")，这也给了我垃圾值，因为它们也具有相同的嵌套结构。有什么方法可以让我只获得头衔吗？最终，我必须从表中获取标题、总总额、打开和关闭列的数据作为每个月和每年的 csv 文件。

Answer 1

首先你需要找到当前表。 您可以看到该内容位于“中心”标签内。 好的，试试：soup.find('center')。当前表是第一个，所以：table = soup.find('center').find_all('table')[0].
现在在此表中尝试查找链接：
trs = table.find_all('tr') trs = iter(trs) next(trs) #skip first element for tr in trs: try: print tr.find_all('a')[0]['href'] print tr.find_all('a')[0].get_text() except: print "can't find a"

Answer 2

您可以使用 lxml 模块（如果编码正确，比 BeautifulSoup 快一个数量级）从提供如下内容的 url 获取数据：

import requests
from lxml import html

url = "http://www.boxofficemojo.com/monthly/?view=releasedate&chart=&month=1&yr=2006"

response = requests.get(url)
soup = html.fromstring(response.content)
result_list = []
for row in soup.xpath('//div[@id="body"]/center/table')[0].xpath('.//tr')[2:] : 
    # print row.xpath()
    data = row.xpath('./td//text()')
    print data
    if len(data) >= 8 :
        print data
        result_list.append({'title' : data[1].strip(), 'gross' : data[3].strip(),
            'open' : data[7].strip(), 'close' : data[8].strip()})

print result_list

这将导致：

[{'close': '6/1', 'gross': '$70,165,972', 'open': '1/27', 'title': "Big Momma's House 2"}, {'close': '3/12', 'gross': '$62,318,875', 'open': '1/20', 'title': 'Underworld: Evolution'}, {'close': '2/16', 'gross': '$47,326,473', 'open': '1/6', 'title': 'Hostel'}, {'close': '5/4', 'gross': '$47,144,110', 'open': '1/27', 'title': 'Nanny McPhee'}, {'close': '5/11', 'gross': '$42,647,449', 'open': '1/13', 'title': 'Glory Road'}, {'close': '3/9', 'gross': '$38,399,961', 'open': '1/13', 'title': 'Last Holiday'}, {'close': '4/13', 'gross': '$17,127,992', 'open': '1/27', 'title': 'Annapolis'}, {'close': '3/30', 'gross': '$14,734,633', 'open': '1/13', 'title': 'Tristan and Isolde'}, {'close': '3/9', 'gross': '$11,967,000', 'open': '1/20', 'title': 'End of the Spear'}, {'close': '6/25', 'gross': '$10,407,978', 'open': '1/27', 'title': 'Roving Mars (IMAX)'}, {'close': '2/23', 'gross': '$6,090,172', 'open': '1/6', 'title': "Grandma's Boy"}, {'close': '1/22', 'gross': '$2,405,420', 'open': '1/6', 'title': 'BloodRayne'}, {'close': '4/6', 'gross': '$2,197,694', 'open': '1/27', 'title': 'Rang De Basanti'}, {'close': '5/18', 'gross': '$1,439,972', 'open': '1/20', 'title': 'Why We Fight'}, {'close': '5/4', 'gross': '$1,253,413', 'open': '1/27', 'title': 'Tristram Shandy: A Cock and Bull Story'}, {'close': '3/9', 'gross': '$888,975', 'open': '1/20', 'title': 'Looking for Comedy in the Muslim World'}, {'close': '3/23', 'gross': '$672,243', 'open': '1/27', 'title': 'Imagine Me and You'}, {'close': '1/29', 'gross': '$332,491', 'open': '1/13', 'title': 'Zinda'}, {'close': '3/9', 'gross': '$274,245', 'open': '1/20', 'title': 'Dirty'}, {'close': '5/4', 'gross': '$196,857', 'open': '1/6', 'title': 'Fateless'}, {'close': '2/23', 'gross': '$145,626', 'open': '1/27', 'title': 'Bubble'}, {'close': '3/23', 'gross': '$78,378', 'open': '1/27', 'title': 'Manderlay'}, {'close': '2/12', 'gross': '$65,429', 'open': '1/20', 'title': 'The Real Dirt on Farmer John'}, {'close': '2/26', 'gross': '$55,398', 'open': '1/13', 'title': 'That Man: Peter Berlin'}, {'close': '8/24', 'gross': '$53,580', 'open': '1/27', 'title': 'La Petite Jerusalem'}, {'close': '2/2', 'gross': '$29,710', 'open': '1/13', 'title': 'Henri Cartier-Bresson: The Impassioned Eye'}, {'close': '4/6', 'gross': '$24,038', 'open': '1/13', 'title': 'When the Sea Rises'}, {'close': '1/16', 'gross': '$20,055', 'open': '1/11', 'title': 'State of Fear'}, {'close': '4/9', 'gross': '$17,341', 'open': '1/13', 'title': 'Film Geek'}, {'close': '3/30', 'gross': '$16,377', 'open': '1/13', 'title': "April's Shower"}, {'close': '1/29', 'gross': '$11,290', 'open': '1/27', 'title': 'Live Freaky! Die Freaky!'}, {'close': '1/26', 'gross': '$5,716', 'open': '1/20', 'title': 'Pizza'}]

大家可以参考scraping和lxml的文档了解更多。

Answer 3

这里另一种（不是“漂亮”的；））方法也适用于标签或页面结构的（错误）变化：

htmlSection = html_source[html_source.find('<a href="/movies/?id=')+21:]
# ^--- skip the first occurence (it doesn't belong to the table of interest)
while htmlSection.find('<a href="/movies/?id=') > 0  :
    htmlSection = htmlSection[ htmlSection.find('<a href="/movies/?id=') : ]
    htmlSection = htmlSection[htmlSection.find('>') : ]
    titleEndPos = htmlSection.find('</a>')
    strTitle = htmlSection[1:titleEndPos]
    print(strTitle)

基于 .find('headerText'/'trailerText') 原则，您还可以仅使用基本的 Python 字符串操作来提取任何其他信息。