【问题标题】:Scrapy BaseSpider: How does it work?Scrapy BaseSpider:它是如何工作的?
【发布时间】:2009-11-27 00:15:37
【问题描述】:

这是 Scrapy 教程中的 BaseSpider 示例:

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector

from dmoz.items import DmozItem

class DmozSpider(BaseSpider):
   domain_name = "dmoz.org"
   start_urls = [
       "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
       "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
   ]

   def parse(self, response):
       hxs = HtmlXPathSelector(response)
       sites = hxs.select('//ul[2]/li')
       items = []
       for site in sites:
           item = DmozItem()
           item['title'] = site.select('a/text()').extract()
           item['link'] = site.select('a/@href').extract()
           item['desc'] = site.select('text()').extract()
           items.append(item)
       return items

SPIDER = DmozSpider()

我复制了它并对我的项目进行了更改:

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from scrapy.item import Item
from firm.items import FirmItem

class Spider1(CrawlSpider):
    domain_name = 'wc2'
    start_urls = ['http://www.whitecase.com/Attorneys/List.aspx?LastName=A']

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        sites = hxs.select('//td[@class="altRow"][1]/a/@href').re('/.a\w+')
        items = []
        for site in sites:
            item = FirmItem
            item['school'] = hxs.select('//td[@class="mainColumnTDa"]').re('(JD)(.*?)(\d+)')
            items.append(item)
        return items

SPIDER = Spider1()    

我得到了错误

[wc2] ERROR: Spider exception caught while processing   
<http://www.whitecase.com/Attorneys/List.aspx?LastName=A> (referer: <None>): 
[Failure instance: Traceback: <type 'exceptions.TypeError'>: 
'ItemMeta' object does not support item assignment

如果这里的专家查看代码并告诉我哪里出错了,我将不胜感激。

谢谢

【问题讨论】:

    标签: python web-crawler scrapy


    【解决方案1】:

    您的意思可能是item = FirmItem() 而不是item = FirmItem

    【讨论】:

    • 注意如果你使用ItemLoader你会得到同样的错误(参数item是Item实例而不是Item类)
    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2021-09-26
    • 2011-03-18
    • 2015-06-16
    • 2012-08-31
    • 2012-07-13
    相关资源
    最近更新 更多