CrawlSpider 只抓取 start_urls

Question

总之，我尝试构建一个蜘蛛，它会爬取 sherdog 网站并给我所有战士的个人资料（姓名、出生日期、身高、国籍）。如果我运行蜘蛛，它会很好地解析 start_urls。问题是 crawlspider 不会开始爬行，所以我最终只得到了 2 个解析项。我阅读了文档，但对 Scrapy 也很陌生，所以我可能会错过一些东西。你知道什么吗？该网站使用相对 url，所以我首先认为这可能是问题，但是在构建绝对 url 之后它仍然没有工作。我真的希望你们能帮助我！

class ProfileSpyder(CrawlSpider):
  name = "Profile"
  allowed_domains = ["http://www.sherdog.com/fighter/"]
  start_urls = ["http://www.sherdog.com/fighter/Daniel-Cormier-52311", 
            "http://www.sherdog.com/fighter/Ronda-Rousey-73073"]

  Rules = (
      Rule(LinkExtractor(allow=('/fighter/')), callback='parse_item', follow=True)      
      )

  def parse_item(self, response):
    #Build absolute urls and sent new requests
    for href in   response.xpath("/html/body/div[3]/div[2]/div[1]/section[2]/div"):
        url = response.urljoin(href.extract())
        yield scrapy.Request(url, callback=self.parse_item)       
    #Parse item                     
    item = FighterProfile()
    item['Name'] =   response.xpath('.//section/div/h1/span[@class="fn"]/text()').extract()
    item['Birthdate'] = response.xpath('.//section/div/div/div/div/div/div/span/span[@itemprop="birthDate"]/text()').extract()
    item['Height'] = response.xpath('.//section/div/div/div/div/div/div/span[@class="item height"]/strong/text()').extract()
    item['Nationality'] = response.xpath('.//section[1]/div/div[1]/div[1]/div/div[1]/div[1]/span[2]/strong/text()').extract()
    yield item 

还有日志：

2015-12-07 18:15:11 [scrapy] INFO: Scrapy 1.0.3 started (bot: ufcfights)
2015-12-07 18:15:11 [scrapy] INFO: Optional features available: ssl, http11, boto
2015-12-07 18:15:11 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'ufcfights.spiders', 'FEED_URI': 'items.csv', 'SPIDER_MODULES': ['ufcfights.spiders'], 'BOT_NAME': 'ufcfights', 'USER_AGENT': 'Chrome/46.0.2490.80', 'FEED_FORMAT': 'csv', 'DOWNLOAD_DELAY': 1.0}
2015-12-07 18:15:12 [scrapy] INFO: Enabled extensions: CloseSpider, FeedExporter, TelnetConsole, LogStats, CoreStats, SpiderState
2015-12-07 18:15:12 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-12-07 18:15:12 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-12-07 18:15:12 [scrapy] INFO: Enabled item pipelines:
2015-12-07 18:15:12 [scrapy] INFO: Spider opened
2015-12-07 18:15:12 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-12-07 18:15:12 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-12-07 18:15:12 [scrapy] DEBUG: Crawled (200) <GET http://www.sherdog.com/fighter/Daniel-Cormier-52311> (referer: None)
2015-12-07 18:15:13 [scrapy] DEBUG: Crawled (200) <GET http://www.sherdog.com/fighter/Ronda-Rousey-73073> (referer: None)
2015-12-07 18:15:14 [scrapy] INFO: Closing spider (finished)
2015-12-07 18:15:14 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 452,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 46874,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 2,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2015, 12, 7, 17, 15, 14, 92000),
 'log_count/DEBUG': 3,
 'log_count/INFO': 7,
 'response_received_count': 2,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'start_time': datetime.datetime(2015, 12, 7, 17, 15, 12, 618000)}
 2015-12-07 18:15:14 [scrapy] INFO: Spider closed (finished)

Answer 1

在使用CrawlSpider 时不能覆盖parse 方法，它在内部使用parse 作为规则。检查警告here

只需更改规则上的回调方法即可。