scrapy爬虫无法从主页工作

Question

我在http://www.shop.ginakdesigns.com/main.sc 上写了一个潦草的涂鸦，试图收集物品

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import Selector

from .. import items

class GinakSpider(CrawlSpider):
    name = "ginak"
    start_urls = [
   "http://www.shop.ginakdesigns.com/main.sc"
    ]
    rules = [Rule(SgmlLinkExtractor(allow=[r'category\.sc\?categoryId=\d+'])),
        Rule(SgmlLinkExtractor(allow=[r'product\.sc\?productId=\d+&categoryId=\d+']), callback='parse_item')]

def parse_item(self, response):
    sel = Selector(response)
    self.log(response.url)
    item = items.GinakItem()
    item['name'] = sel.xpath('//*[@id="wrapper2"]/div/div/div[1]/div/div/div[2]/div/div/div[1]/div[1]/h2/text()').extract()
    item['price'] = sel.xpath('//*[@id="listPrice"]/text()').extract()
    item['description'] = sel.xpath('//*[@id="wrapper2"]/div/div/div[1]/div/div/div[2]/div/div/div[1]/div[4]/div/p/text()').extract()
    item['category'] = sel.xpath('//*[@id="breadcrumbs"]/a[2]/text()').extract()

    return item

但是它不会超出主页进入任何链接。我已经尝试了各种方法并检查了 SgmlLinkExtractor 的正则表达式。这里有什么问题吗？

Answer 1

问题是您尝试提取的链接中插入了jsessionid，例如：

<a href="/category.sc;jsessionid=EA2CAA7A3949F4E462BBF466E03755B7.m1plqscsfapp05?categoryId=16">

通过对任何字符使用.*?非贪婪匹配而不是寻找/?来修复它：

rules = [Rule(SgmlLinkExtractor(allow=[r'category\.sc.*?categoryId=\d+']), callback='parse_item'),
         Rule(SgmlLinkExtractor(allow=[r'product\.sc.*?productId=\d+&categoryId=\d+']), callback='parse_item')]

希望对您有所帮助。