Beautiful Soup 会在“”和“<”等特殊字符上崩溃

Question

我正在尝试使用漂亮的汤来抓取基于原子的 RSS 提要，但事实证明这很困难。捕获数据一直很好，直到出现<item> 破坏代码并使脚本崩溃。这样的<item>s 始终有标签（Firefox 将它们标记为橙色），如“& lt;”或“& quot;”，而没有它们的 s 可以正常工作。我已经尝试了很多东西，比如 BeautifulStoneSoup，用正则表达式去除特殊字符，并设置“xml”参数，但没有任何效果，而且它们通常只是发出关于在 BS4 中被弃用的警告。

为什么会出现这些字符，我该如何有效地处理它们？

这是我要抓取的页面： http://www.thestar.com/feeds.articles.news.gta.rss

这是我的代码：

news_url = "http://www.thestar.com/feeds.articles.news.gta.rss" # Toronto Star RSS Feed

try:    
    news_rss = urllib2.urlopen(news_url)
    news = news_rss.read()
    news_rss.close()
    soup = BeautifulSoup(news)
except:
    return "error"

titles = soup.findAll('title')
links = soup.findAll('link')

for link in links:
    link = link.contents    # I want the url without the <link> tags

news_stuff = []
for item in titles:
    if item.text == "TORONTO STAR | NEWS | GTA":    # These have <title> tags and I don't want them; just skip 'em.
        pass
    else:
        news_stuff.append((item.text, links[i]))    # Here's a news story.  Grab it.

i = 0
for thing in news_stuff:
    print '<a href="' 
    print thing[1]
    print '"target="_blank">' 
    print thing[0]
    print '</a><br/>'
    i += 1

Answer 1

不确定你说的是哪个问题，但是我在运行你的代码时遇到了这个错误：

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 54: ordinal not in range(128)

为了解决这个问题，我刚刚添加了编码：

for thing in news_stuff:
    print '<a href="' 
    print thing[1]
    print '"target="_blank">' 
    print thing[0].encode("utf-8")
    print '</a><br/>'
    i += 1

该脚本执行后没有任何错误。

Answer 2

这是我尝试过的，它没有崩溃。

from string import punctuation, whitespace
import urllib2
import datetime
import re
import MySQLdb
import csv
from bs4 import BeautifulSoup as Soup
news_url = "http://www.thestar.com/feeds.articles.news.gta.rss" # Toronto Star RSS Feed

news_rss = urllib2.urlopen(news_url)
news = news_rss.read()
news_rss.close()
soup = Soup(news)

titles = soup.findAll('title')
links = soup.findAll('link')

for link in links:
    link = link.contents    # I want the url without the <link> tags
i=0
news_stuff = []
for item in titles:
    if item.text == "TORONTO STAR | NEWS | GTA":    # These have <title> tags and I don't want them; just skip 'em.
        pass
    else:
        news_stuff.append((item.text, links[i]))    # Here's a news story.  Grab it.

i = 0
for thing in news_stuff:
    print '<a href="' 
    print thing[1]
    print '"target="_blank">' 
    print thing[0]
    print '</a><br/>'
    i += 1

这是我得到的输出

<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
TTC argues for return of special constables
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Health information of 18,000 people stolen in Peel Region
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Fire closes Bathurst St. south of Dupont
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Empty tanker train cars derail in Brampton
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Medical illustration studios flourish in Toronto
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
In Texas, Toronto music leaders urge city hall to say ‘yes’
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Making sense of the Sammy Yatim shooting: Fiorito
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Toronto’s chief planner, Jennifer Keesmaat, challenges Mirvish/Gehry scheme: Hume
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Westbound Gardiner lanes reopen after rollover near Spadina
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Daycare Crisis: Halton health complaints show gaps in unlicensed care
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Witness describes shooting details as man confronted police near van
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Muslim AIDS activist honoured for taboo-busting work
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Death to death with dignity: DiManno
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Rockers join forces in Line 9 protest
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Could you eat 10 pizzas in 12 minutes? This guy did
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Former participants speak up about gay healing program
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Freed Canadians Tarek Loubani and John Greyson awaiting papers to come home from Egypt
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Man dies after crash at Finch and Dufferin
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Nuit Blanche lights up Toronto Saturday night
</a><br/>
<a href="
<link>http://www.thestar.com/feeds.articles.news.gta.rss</link>
"target="_blank">
Leafs fans celebrate home opener at Maple Leaf Square
</a><br/>