将scrapy用于基于多个搜索选项的网站

Question

我是 Scrapy 和 Python 的新手。

我想抓取一个使用基于查询的搜索的房产注册商的网站。我见过的大多数示例都使用简单的网页，而不是通过 FormRequest 机制使用搜索。我写的代码如下。目前所有内容都是硬编码的。我希望能够抓取年份或县的数据库。

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from scrapy.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector

class SecondSpider(CrawlSpider):
    name = "second"

    '''
    def start_requests(self):
        return [scrapy.FormRequest("https://www.propertypriceregister.ie/website/npsra/pprweb.nsf/PPR?OpenForm"# this is the form here it asks for the following,
                                    # then the linke changes to this form
                                   https://www.propertypriceregister.ie/website/npsra/PPR/npsra-ppr.nsf/PPR-By-Date?SearchView
                                   &Start=1
                                   &SearchMax=0
                                   &SearchOrder=4
                                   &Query=%5Bdt_execution_date%5D%3E=01/01/2010%20AND%20%5Bdt_execution_date%5D%3C01/01/2011
                                   &County=                 # this are the fields of query
                                   &Year=2010               # this are the fields of query
                                   &StartMonth=             # this are the fields of query
                                   &EndMonth=               # this are the fields of query
                                   &Address=                # this are the fields of query

                                   formdata={'user': 'john', 'pass': 'secret'},
                                   callback=self.logged_in)]

    def logged_in(self, response):
        # here you would extract links to follow and return Requests for
        # each of them, with another callback
        pass
    '''
    allowed_domains = ["www.propertypriceregister.ie"]
    start_urls = ('https://www.propertypriceregister.ie/website/npsra/pprweb.nsf/PPR?OpenForm',)

    rules = (
        Rule(SgmlLinkExtractor(allow='/website/npsra/PPR/npsra-ppr.nsf/PPR-By-Date?SearchView&Start=1&SearchMax=0&SearchOrder=4&Query=%5Bdt_execution_date%5D%3E=01/01/2010%20AND%20%5Bdt_execution_date%5D%3C01/01/2011&County=&Year=2010&StartMonth=&EndMonth=&Address='),
            callback='parse',
            follow= True),
    )

    def parse(self, response):
        print response
        pass

Answer 1

开始之前，请重新阅读Rule 对象的工作原理。目前，您的规则将匹配网站永远不会显示链接的非常具体的 URL（因为它采用表单帖子的格式）。

接下来，不要覆盖 CrawlSpider 的 parse 函数（实际上，根本不要使用它）。 CrawlSpider 在内部使用它来运行（有关更多详细信息，请参阅我提供的链接上的警告）。

您需要为每个要调用的元素生成一个FormRequest，类似于这样（注意：未经测试，但它应该可以工作）：

import itertools
... # all your other imports here

class SecondSpider(CrawlSpider):
    name = 'second'
    allowed_domains = ['propertypriceregister.ie', 'www.propertypriceregister.ie']

    rules = (
        Rule(LinkExtractor(allow=("/eStampUNID/UNID-")), callback='parse_search'),
    )

    def start_requests(self):
        years = [2010, 2011, 2012, 2013, 2014]
        counties = ['County1', 'County2')
        for county, year in itertools.product(*[counties, years]):
            yield scrapy.FormRequest("https://www.propertypriceregister.ie/website/npsra/pprweb.nsf/PPR?OpenForm",
                                      formdata={'County': county, 'Year': year}, 
                                      dont_filter=True)

    def parse_search(self, response):
        # Parse response here

从此时起，您的规则将应用于您从 FormRequest 返回的每个页面，以从中提取 URL。如果您想真正从这些初始 url 中获取任何内容，请覆盖 CrawlSpider 的 parse_start_url 方法。