【问题标题】:Missing class in beautifulsoupBeautifulsoup 中的缺课
【发布时间】:2016-09-16 04:19:35
【问题描述】:

我正在尝试获取 MTA 信息页面上的 div。当我抓取 html 并用 BeautifulSoup 解析它时,它似乎丢失了一些数据。

这是我目前的代码

from bs4 import BeautifulSoup
import urllib # access the web

# SUBWAY STATUS PROJECT
userURL = "http://www.mta.info" # MTA SITE

htmlfile = urllib.urlopen(userURL) #creates html file
htmldoc = htmlfile.read()   #creates html text

soup = BeautifulSoup(htmldoc, 'html.parser')    

subChart = soup.find( id = 'subwayDiv')

print subChart

我使用 print 只是为了确保我获得了所有数据。我发现我遗漏了一些我想要获取的信息。如果我自己查看该页面,我会发现我缺少一个显示地铁状态的类的 div。

我对编程很陌生,所以请注意我的无知

【问题讨论】:

  • 它们是由ajax创建的,不是普通的静态html,所以尝试另一种方式。

标签: python html parsing beautifulsoup


【解决方案1】:

数据是通过 ajax 请求检索的,您可以获取 json 格式的信息,您唯一需要将其传递给您可以获取的 timestamp time.time() 然后用json 库解析它:

from time import time
from json import load, loads
import urllib

url  = "http://www.mta.info/service_status_json/{}".format(int(time()))

json_dict = loads(load(urllib.urlopen(url))) 

from pprint import pprint as pp
pp(json_dict)

我不会添加所有输出,因为输出太多,但使用 "BT" 我们得到:

{u'line': [{u'Date': {},
            u'Time': {},
            u'name': u'Bronx-Whitestone',
            u'status': u'GOOD SERVICE',
            u'text': {}},
           {u'Date': {},
            u'Time': {},
            u'name': u'Cross Bay',
            u'status': u'GOOD SERVICE',
            u'text': {}},
           {u'Date': {},
            u'Time': {},
            u'name': u'Henry Hudson',
            u'status': u'GOOD SERVICE',
            u'text': {}},
           {u'Date': u'09/16/2016',
            u'Time': u' 5:57AM',
            u'name': u'Hugh L. Carey',
            u'status': u'SERVICE CHANGE',
            u'text': u"                    <span class='TitleServiceChange' >Service Change</span>                    <span class='DateStyle'>                    &nbsp;Posted:&nbsp;09/16/2016&nbsp; 5:57AM                    </span><br/><br/>                  HLC - HOV Lane Open 6 AM to 10 AM. Two-Way Operations in effect. Three (3) lanes Manhattan-bound. One (1) lane Brooklyn-bound.                <br/><br/>              "},
           {u'Date': {},
            u'Time': {},
            u'name': u'Marine Parkway',
            u'status': u'GOOD SERVICE',
            u'text': {}},
           {u'Date': u'09/16/2016',
            u'Time': u' 5:57AM',
            u'name': u'Queens Midtown',
            u'status': u'SERVICE CHANGE',
            u'text': u"                    <span class='TitleServiceChange' >Service Change</span>                    <span class='DateStyle'>                    &nbsp;Posted:&nbsp;09/16/2016&nbsp; 5:57AM                    </span><br/><br/>                  QMT - HOV Lane Open 6 AM to 10 AM. Two-Way Operation in effect. Three (3) lanes Manhattan bound. One (1) lane Queens bound.                <br/><br/>                                  <span class='TitlePlannedWork' >Planned Work</span>                    <br/>                  <P  style='MARGIN: 0in 0in 0pt'><SPAN style=''Times New Roman';2016; Queens-Midtown Tunnel downtown exit; One lane closed. Use 37<SUP>th</SUP></FONT><FONT size=3> St tunnel exit for access to 2</FONT><SUP><FONT size=3>nd</FONT></SUP><FONT size=3> Ave. Motorists should allow extra time and may wish to use an alternate route if possible' Drivers should expect delays and plan accordingly. Motorists can sign up for MTA e-mail or text alerts at </FONT><SPAN style='COLOR: blue'><A href='http://www.mta.info/'><SPAN style='COLOR: #0563c1'><FONT size=3>www.mta.info</FONT></SPAN></A><FONT size=3> </FONT></SPAN><FONT size=3>and check the Bridges and Tunnels homepage or Facebook page for the latest information on this planned work.</FONT></FONT></SPAN></P>                <br/><br/>                                  <span class='TitlePlannedWork' >Planned Work</span>                    <br/>                  QMT- MANHATTAN PLAZA WORK REQUIRES CLOSURE OF 'CROSSTOWN' LANES FOR 2 MONTHS. CUSTOMERS SEEKING A CROSSTOWN MANHATTAN ROUTE USE THE UPTOWN LANES; EXPECT DELAYS.                <br/><br/>              "},
           {u'Date': u'08/15/2016',
            u'Time': u' 3:56PM',
            u'name': u'Robert F. Kennedy',
            u'status': u'PLANNED WORK',
            u'text': u"                    <span class='TitlePlannedWork' >Planned Work</span>                    <br/>                  <P  style='MARGIN: 0in 0in 0pt'><SPAN style='COLOR: #1f497d'><FONT size=3 face=Calibri>Starting Monday, August 15, 2016 and through early 2018, one lane will be closed on the Queens-to-Manhattan ramp at the Robert F. Kennedy Bridge for roadway rehabilitation. In addition, overnight on Thursday, August 18 and Friday, August 19, there will be a series of intermittent FULL ramp closures, lasting 15-20 minutes each.</FONT></SPAN></P>                <br/><br/>              "},
           {u'Date': {},
            u'Time': {},
            u'name': u'Throgs Neck',
            u'status': u'GOOD SERVICE',
            u'text': {}},
           {u'Date': u'09/16/2016',
            u'Time': u' 5:28AM',
            u'name': u'Verrazano-Narrows',
            u'status': u'PLANNED WORK',
            u'text': u"                    <span class='TitlePlannedWork' >Planned Work</span>                    <br/>                  VNB: PLANNED WORK; S. I. BOUND LOWER LEVEL - ONE LANE CLOSED; EXPECT DELAYS.                <br/><br/>              "}]}

所以你只需要通过字典并挑选出你想要的。

【讨论】:

  • 谢谢,我回家后试试瘦身!
【解决方案2】:

在 subchart 变量中查找具有类 metroCategory 的元素并存储属性 id 的值。 例如:来自这部分数据

<div style="float: left; width: 220px; border-bottom: 1px solid #7B7B98; padding: 4px 0;">
<div class="span-11"><img alt="1 2 3 Subway" class="subwayIcon_123" src="http://www.mta.info/sites/all/modules/custom/servicestatus/images/img_trans.gif"/></div>
<div class="subwayCategory" id="123" style="margin-top: 4px;"></div>

地铁类别的 div 的 id 值为 123。 现在向http://www.mta.info/status/subway/{ID}提出请求

将术语{ID} 替换为您想要的ID

【讨论】:

  • 那行不通。在您的浏览器或代码中尝试一下。
猜你喜欢
  • 2022-01-04
  • 2017-01-20
  • 1970-01-01
  • 2016-03-12
  • 2017-06-17
  • 2021-04-03
  • 1970-01-01
  • 2022-08-18
  • 2017-12-02
相关资源
最近更新 更多