如何在抓取网页时单击下一步按钮答案

【问题标题】：How to click next button while scraping a webpage如何在抓取网页时单击下一步按钮
【发布时间】：2019-10-09 00:12:23
【问题描述】：

我正在使用具有多页信息的 scrapy 抓取网页，我需要程序单击下一步按钮，然后抓取下一页，然后继续这样做，直到所有页面都被抓取。但我不知道该怎么做，我只能刮第一页。

from scrapy_splash import SplashRequest
from ..items import GameItem

class MySpider(Spider):
        name = 'splash_spider' # Name of Spider
        start_urls = ['http://www.starcitygames.com/catalog/category/10th%20Edition'] # url(s)
        def start_requests(self):
                for url in self.start_urls:
                        yield SplashRequest(url=url, callback=self.parse, args={"wait": 3})
        #Scraping
        def parse(self, response):
                item = GameItem()
                for game in response.css("tr"):
                        # Card Name
                        item["Name"] = game.css("a.card_popup::text").extract_first()
                        # Price
                        item["Price"] = game.css("td.deckdbbody.search_results_9::text").extract_first()
                        yield item

【问题讨论】：

你为什么使用 Splash？你看过stackoverflow.com/q/8550114/939364吗？

标签： python web-scraping scrapy splash-screen

【解决方案1】：

你可以让它像下面这样工作：

import scrapy

class GameSpider(scrapy.Spider):
        name = 'game_spider'
        start_urls = ['http://www.starcitygames.com/catalog/category/10th%20Edition'] 

        def parse(self, response):
                for game in response.css("tr.deckdbbody_row"):
                        yield {
                                'name': game.css("a.card_popup::text").get(),
                                'price': game.css(".search_results_9::text").get(),
                        }
                next_page_url = response.css('table+ div a:nth-child(8)::attr(href)').get()
                if next_page_url is not None:
                        yield response.follow(next_page_url, self.parse)

【讨论】：

【解决方案2】：

您可以使用 css 选择器来定位下一个按钮

a:nth-of-type(7)

与第 n 个孩子

a:nth-child(8)

【讨论】：

【解决方案3】：

Documentation 非常明确：

from scrapy_splash import SplashRequest
from ..items import GameItem

class MySpider(Spider):
        name = 'splash_spider' # Name of Spider
        start_urls = ['http://www.starcitygames.com/catalog/category/10th%20Edition'] # url(s)
        def start_requests(self):
            for url in self.start_urls:
                yield SplashRequest(url=url, callback=self.parse, args={"wait": 3})
        #Scraping
        def parse(self, response):
            item = GameItem()
            for game in response.css("tr"):
                # Card Name
                item["Name"] = game.css("a.card_popup::text").extract_first()
                # Price
                item["Price"] = game.css("td.deckdbbody.search_results_9::text").extract_first()
                yield item

            next_page = response.css(<your css selector to find next page>).get()
            if next_page is not None:
                yield response.follow(next_page, self.parse)

【讨论】：