用 BeautifulSoup 刮痧

Question

我在抓取网页中的某些字段时遇到了一些麻烦。我在下面有一些代码可以很好地执行前两个 for 循环，但我在最后一个 for 循环中遇到了困难。

from bs4 import BeautifulSoup
import urllib2
url="https://www.mturk.com/mturk/findhits?match=false"
page=urllib2.urlopen(url)
soup = BeautifulSoup(page.read())


requesters=soup.findAll('span',{'class':'requesterIdentity'})
for eachrequester in requesters:
   print "Requester Name: "+eachrequester.string

rewards=soup.findAll('span',{'class':'reward'})
for eachreward in rewards:
   print "Reward: "+eachreward.string


hitnames=soup.findAll('a',{'class':'capsulelink'})  #THE ISSUE IS IN THESE 3 LINES
for eachhitname in hitnames:
   print "Hit Name: "+eachhitname.string

当前输出的代码：

Requester Name: Andrew Ryan
Requester Name: Vishwanath Kumar
Requester Name: rohzit0d
Requester Name: Jon Brelig
Requester Name: Tagasauris
Requester Name: Tagasauris
Requester Name: Tagasauris
Requester Name: CopyText Inc.
Requester Name: Tagasauris
Requester Name: Amazon Requester Inc.
Reward: $0.24
Reward: $0.03
Reward: $0.00
Reward: $0.05
Reward: $0.04
Reward: $0.02
Reward: $0.02
Reward: $0.01
Reward: $0.04
Reward: $0.00

Traceback (most recent call last):
  File "C:/Users/admin/Desktop/pythonimageret/hitgwt.py", line 19, in <module>
    print "Hit Name: "+eachhitname.string
TypeError: cannot concatenate 'str' and 'NoneType' objects

我意识到脚本在这里找不到 html 的内容。 HTML 看起来像：

<a class="capsulelink" href="#" id="capsule6-0">
    Indoors or Out?
    <span class="tags"></span>
</a>

我认为这是因为href="#" id="capsule6-0" 介于class="" 和> 之间

Answer 1

eachhitname.string 是None，因为该属性仅在当前元素中没有其他标签时才包含文本。每个链接都有一个<span class="tags"></span> 元素。

改用.text 属性；添加str.strip()call 以删除多余的空格：

hitnames = soup.findAll('a', {'class':'capsulelink'})
for eachhitname in hitnames:
   print "Hit Name: " + eachhitname.text.strip()