scrapy - 如果跟随一个无限的网站，则终止抓取

Question

假设我有一个像this 这样的网页。

counter.php

if(isset($_GET['count'])){
    $count = intval($_GET['count']);
    $previous = $count - 1;
    $next = $count + 1;
    ?>
    <a href="?count=<?php echo $previous;?>">< Previous</a>

    Current: <?php echo $count;?>

    <a href="?count=<?php echo $next;?>">Next ></a>
    <?
}

?>

这是一个“无限”的网站，因为您可以一直点击下一步进入下一页（计数器只会增加）或上一个等。

但是，如果我想像这样使用 scrapy 抓取此页面并跟踪链接，scrapy 将永远不会停止抓取。

示例蜘蛛：

urls = []  
class TestSpider(CrawlSpider):
        name = 'test'
        allowed_domains = ['example.com']
        start_urls = ['http://example.com/counter?count=1']


        rules = (
            Rule(LinkExtractor(), callback='parse_item', follow=True),
            )

        def parse_item(self, response):
            urls.append(response.url)

我可以使用什么样的机制来确定我是否确实被困在一个无限的网站中并需要摆脱它？

Answer 1

如果该页面上没有ITEMS，或者没有NEXT PAGE按钮，您总是可以突破，这意味着分页已经结束

class TestSpider(CrawlSpider):
        name = 'test'
        allowed_domains = ['example.com']

        def start_requests(self):
            page = 1
            yield Request("http://example.com/counter?page=%s" % (page), meta={"page": page}, callback=self.parse_item)

        def parse_item(self, response):

            #METHOD 1: check if items availble on this page         
            items = response.css("li.items")

            if items:
                #Now go to next page
                page = int(response.meta['page']) + 1
                yield Request("http://example.com/counter?page=%s" % (page), meta={"page": page}, callback=self.parse_item)
            else:
                logging.info("%s was last page" % response.url)

            #METHOD 2: check if this page has NEXT PAGE button, most websites has that          
            nextPage = response.css("a.nextpage")

            if nextPage:
                #Now go to next page
                page = int(response.meta['page']) + 1
                yield Request("http://example.com/counter?page=%s" % (page), meta={"page": page}, callback=self.parse_item)
            else:
                logging.info("%s was last page" % response.url)

Answer 2

您不必在scrapy 中使用Rule。您可以先逐页解析，然后迭代每个页面中的所有项目。或者您可以收集每个页面中的所有项目链接。 例如：

urls = []
class TestSpider(CrawlSpider):
    name = 'test'
    allowed_domains = ['example.com']
    start_urls = ['http://example.com/counter?count=1']

    def parse(self, response):
        links = response.xpath('//a[@class="item"]/@href').extract()
        for link in links:
            yield Request(link, self.parse_item)
            # you can insert the item 's url here, so you dont have to yield to parse_item
            # urls.append(link)

        url, pg = response.url.split("=")# you can break infinite loop here
        if int(pg) <= 10: #We loop by page #10
            yield Request(url + "=" + str(int(pg) + 1), self.parse)

    def parse_item(self, response):
        urls.append(response.url)