爬虫未爬取第二个 start_url答案

【问题标题】：crawler is not crawling second start_url爬虫未爬取第二个 start_url
【发布时间】：2020-04-06 06:38:43
【问题描述】：

我正在尝试从hindustantimes.com 上抓取电影评论和电视新闻。当我运行此代码时，它只会抓取第一个 start_url，但无法抓取第二个 start_url。我认为必须重置计数器，但我不知道如何重置。我想从两个 start_url 中抓取 n 个页面。

import scrapy
#test_push
from..items import HindustantimesItem

class HindustantimesSpider(scrapy.Spider):
    name = 'Hindustantimes_review'
    page_number = 2
    count = 0

    def start_requests(self): 

        urls = ['https://www.hindustantimes.com/movie-reviews/page/?pageno={}',
        'https://www.hindustantimes.com/tv/page/?pageno={}',
        ]
        ur = []
        for url in urls:
            for i in range(1,3):
                x = url.format(i)
                yield scrapy.Request(url=x, callback=self.parse,)

    def parse(self, response):
        print("-------^^^^^^---------")
        print(response.request.url)
        items = {}


        with open('output.txt', 'a') as the_file:
            the_file.write(response.request.url)
            the_file.write( "\n")

        title_xpath = ['//*[@id="scroll-container"]/ul/li[{}]/div/div[2]/div/a/text()', '/html/body/div[1]/section/div[2]/div/div[1]/div[2]/ul/li[{}]/div/div[2]/div[1]/a/text()']
        page_review_xpath = ['//*[@id="scroll-container"]/ul/li[{}]/div/div[2]/p/text()','/html/body/div[1]/section/div[2]/div/div[1]/div[2]/ul/li[{}]/div/div[2]/div[2]/text()']
        page_link_xpath = ['//*[@id="scroll-container"]/ul/li[{}]/div/div[2]/div/a/@href', '/html/body/div[1]/section/div[2]/div/div[1]/div[2]/ul/li[{}]/div/div[2]/div[1]/a/@href']
        if HindustantimesSpider.count ==0:
            current_title_xpath = title_xpath[0]
            current_review_xpath = page_review_xpath[0]
            current_link_xpath = page_link_xpath[0]
            HindustantimesSpider.count+=1
        else:
            current_title_xpath = title_xpath[1]
            current_review_xpath = page_review_xpath[1]
            current_link_xpath = page_link_xpath[1]

        count = response.xpath(current_title_xpath.format("*")).getall()
        count = len(count)
        i = 1
        while i<=count:
            outputs = HindustantimesItem()
            outputs['page_title'] = response.xpath(current_title_xpath.format(i)).get()
            outputs['review_content'] = response.xpath(current_review_xpath.format(i)).get()
            outputs['review_link'] = response.xpath(current_link_xpath.format(i)).get()
            i+=1
            fl = 0
            if outputs['page_title'] == []:
                outputs['page_title'] = ''
                fl+=1
            if outputs['review_content'] == []:
                outputs['review_content'] = ''
                fl+=1
            if outputs['review_link'] == []:
                outputs['review_link'] = ''
                fl += 1
            else:
                if 'Review:' in outputs['page_title'].split(" ") or 'review:' in outputs['page_title'].split(" "):
                    yield outputs

        pass

【问题讨论】：

开始爬取时的输出是什么？
第 1 页输出没有出现，因为我将两个 xpath 第一个用于 url 的第一页，第二个 xpath 用于两个 url 的第 2 页起
从您的代码中可以看出，其中有几个问题。当蜘蛛开始爬行时，它会在输出中打印事件。阅读输出可以帮助您了解您的代码问题并帮助我们了解那里发生了什么。

标签： python python-3.x xpath scrapy web-crawler

【解决方案1】：

def start_requests(self): 

        urls = ['https://www.hindustantimes.com/movie-reviews/page/?pageno={}',
        'https://www.hindustantimes.com/tv/page/?pageno={}',
        ]
        ur = []
        for url in urls:
            for i in range(1,3):
                x = url.format(i)
                yield scrapy.Request(url=x, callback=self.parse,dont_filter=True)

【讨论】：

参见第 1 页的 url 有不同的 xpath，从第 2 页开始，它有不同的 xpath。并且我从第 2 页获取输出没有得到第 1 页输出。