Web Scraping Python Access Table 数据

Question

所以我正在尝试使用 Beautiful Soup 对该网站http://www.killedbypolice.net/kbp2013.html 进行一些网络抓取并访问表中的数据。我当前的代码是：

url = "http://www.killedbypolice.net/kbp2013.html"
page = urllib.request.urlopen(url)
soup = BeautifulSoup(page, "html.parser")

data = soup.find_all('table')
data[0]

但是...我遇到了最大递归深度运行时错误。我不确定如何访问保存数据的表内的“td”字段。谢谢

Answer 1

错误是因为 html 的格式非常糟糕，您会得到 RuntimeError: maximum recursion depth exceeded 使用 lxml 和 html.parser 创建汤对象，唯一有效的解析器是 html5lib：

html = requests.get("http://www.killedbypolice.net/kbp2013.html").content
soup = BeautifulSoup(html, "html5lib")

# get all the table rows
table = soup.find("table")

这得到了桌子：

    <table background="/kbp/bg1.jpg" border="1" cellpadding="0" cellspacing="0" width="100%">
<tbody><tr><td><img src="http://www.killedbypolice.net/size.jpg" width="185"/><br/><b><center># since Jan 1st '14</center></b></td>
<td><b><center>St.</center></b></td>
<td><b><center>g/r</center></b></td>
<td><img src="http://www.killedbypolice.net/size.jpg" width="200"/><b><center>Name, Age</center></b></td>
<td></td>
<td><img src="http://www.killedbypolice.net/size.jpg" width="297"/><b><center>KBP link <font color="red">(plus extensive follow-ups)</font></center></b></td>
<td><b><center>News link</center></b></td>
......................................................................
</font></a></td></tr><tr><td><center>(2) May 2, 2013        </center>
</td><td>CA</td><td>M/B</td><td>Kenneth Bernard Williams, 55   </td><td><font size="2">G</font></td><td><a href="http://facebook.com/KilledByPolice/posts/622539181107556" target="new"><font size="2"></font></a><font size="2"><center><a href="http://facebook.com/KilledByPolice/posts/622539181107556" target="new">facebook.com/KilledByPolice/posts/622539181107556  </a></center></font></td><td><a href="http://www.nbclosangeles.com/news/local/Police-Shoot-Kill-Suspect-in-Skid-Row-Prompting-Angry-Crowd-to-Gather-205646861.html" target="new"><font size="2">http://www.nbclosangeles.com/news/local/Police-Shoot-Kill-Suspect-in-Skid-Row-Prompting-Angry-Crowd-to-Gather-205646861.html
</font></a></td></tr><tr><td><center>(1) May 1, 2013        </center></td><td>MI</td><td>M/B</td><td><a href="http://www.killedbypolice.net/victims/2261.jpg" target="new">Jordan West-Morson, 26   </a></td><td><font size="2">G</font></td><td><a href="http://facebook.com/KilledByPolice/posts/1033800406648096" target="new"></a><center><a href="http://facebook.com/KilledByPolice/posts/1033800406648096" target="new"><font size="2">facebook.com/KilledByPolice/posts/1033800406648096    </font></a></center></td><td><a href="http://www.mlive.com/news/detroit/index.ssf/2013/09/detroit_transit_officer_charge.html" target="new"><font size="2">http://www.mlive.com/news/detroit/index.ssf/2013/09/detroit_transit_officer_charge.html</font></a><font size="2"><br/><i>Detroit transit officer not guilty in fatal shooting: </i><a href="http://www.clickondetroit.com/news/detroit-transit-officer-not-guilty-in-fatal-shooting/32405878" target="new">http://www.clickondetroit.com/news/detroit-transit-officer-not-guilty-in-fatal-shooting/32405878

</a></font></td></tr></tbody></table>

然后是对 find_all 的简单调用：

print(table.find_all("tr"))

给你：

 AttributeError: 'NoneType' object has no attribute 'next_element'

html 完全是一团糟，不幸的是我看不到用 bs4 解析它的简单方法，这可能是您需要使用一些正则表达式的罕见情况之一。