如何从 CSV 导入 start_urls

Question

我在 scrapy 中有一个程序可以抓取多个 URL，每个 URL 都有多个页面，现在它正在输出乱序。它将输出所有页面 1，然后是所有页面 2，依此类推，而不是 URL 1 的页面 1,2.3... 和 URL 2 的页面 1,2,3 等等。无论如何，经过大量研究后，解决此问题的方法似乎是从我的 CSV 导入我的 URL，然后将其设置为等于 start_urls，但我无法弄清楚如何在我当前的程序中做到这一点。

# Import from other python files and scrapy files and the needed csv file containing all URLs/proxies/ua
import csv
import scrapy
from scrapy.spiders import Spider
from scrapy_splash import SplashRequest
from ..items import GameItem
from random import randint
from time import sleep
from scrapy.http import FormRequest
##########################          SPALSHSPIDER.PY OVERVIEW      #####################################################
# process the csv file so the url + ip address + useragent pairs are the same as defined in the file
# returns a list of dictionaries, example:
# [ {'url': 'http://www.starcitygames.com/catalog/category/Rivals%20of%20Ixalan',
#    'ip': 'http://204.152.114.244:8050',
#    'ua': "Mozilla/5.0 (BlackBerry; U; BlackBerry 9320; en-GB) AppleWebKit/534.11"},
#    ...
# ]
# plus python file also scrapes all URLs returning needed info and goes to all apges associated with URL by clicking next button

# Function to read csv file that contains URLs that are paried with proxies and user agents
def process_csv(csv_file):
    # Initialize data
    data = []
    # Initialize reader
    reader = csv.reader(csv_file)
    next(reader)

    # While inside csv file and not at end of csv file
    for fields in reader:

        # Set URL
        if fields[0] != "":
            url = fields[0]
        else:
            continue # skip the whole row if the url column is empty
        #Set proxy and pair with correct URL
        if fields[1] != "":
            ip = "http://" + fields[1] + ":8050" # adding http and port because this is the needed scheme
        # Set user agent and pair with correct URL
        if fields[2] != "":
            useragent = fields[2]
        # Put all three together
        data.append({"url": url, "ip": ip, "ua": useragent})
    # Return URL paried with ua and proxy
    return data

# Spider clascrapy crawl splash_spider -o data.json
class MySpider(Spider):

    # Name of Spider
    name = 'splash_spider'
    # getting all the url + ip address + useragent pairs then request them
    def start_requests(self):

        # get the file path of the csv file that contains the pairs from the settings.py
        with open(self.settings["PROXY_CSV_FILE"], mode="r") as csv_file:
           # requests is a list of dictionaries like this -> {url: str, ua: str, ip: str}
            requests = process_csv(csv_file)
        for req in requests:
            # Return needed url with set delay of 3 seconds
            yield SplashRequest(url=req["url"], callback=self.parse, args={"wait": 3},
                    # Pair with user agent specified in csv file
                    headers={"User-Agent": req["ua"]},
                    # Sets splash_url to whatever the current proxy that goes with current URL  is instead of actual splash url
                    splash_url = req["ip"],
                    )

    # Scraping function that will scrape URLs for specified information
   # import pdb; pdb.set_trace()
    def parse(self, response):

        # Initialize item to function GameItem located in items.py, will be called multiple times
        item = GameItem()
        # Initialize saved_name
        saved_name = ""
        # Extract card category from URL using html code from website that identifies the category.  Will be outputted before rest of data

        item["Category"] = response.css("span.titletext::text").get()

        # For loop to loop through HTML code until all necessary data has been scraped
        for game in response.css("tr[class^=deckdbbody]"):

            # Initialize saved_name to the extracted card name
            saved_name  = game.css("a.card_popup::text").get() or saved_name
            # Now call item and set equal to saved_name and strip leading '\n' from output
            item["Card_Name"] = saved_name.strip()
            # Check to see if output is null, in the case that there are two different conditions for one card
            if item["Card_Name"] != None:
                # If not null than store value in saved_name
                saved_name = item["Card_Name"].strip()
            # If null then set null value to previous card name since if there is a null value you should have the same card name twice
            else:
                item["Card_Name"] = saved_name
            # Call item again in order to extract the condition, stock, and price using the corresponding html code from the website
            item["Condition"] = game.css("td[class^=deckdbbody].search_results_7 a::text").get()
            item["Stock"] = game.css("td[class^=deckdbbody].search_results_8::text").get()
            item["Price"] = game.css("td[class^=deckdbbody].search_results_9::text").get()
            if item["Price"] == None:
                item["Price"] = game.css("td[class^=deckdbbody].search_results_9 span[style*='color:red']::text").get()
            # Return values
            yield item

        # Finds next page button
        next_page = response.xpath('//a[contains(., "- Next>>")]/@href').get()
        # If it exists and there is a next page enter if statement
        if next_page is not None:
            # Go to next page
            yield response.follow(next_page, self.parse)

部分 CSV 文件

URL,IP Address,User Agent
http://www.starcitygames.com/catalog/category/Vanguard%20Oversized,199.89.192.78,"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-en) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4"
http://www.starcitygames.com/catalog/category/Visions,,
http://www.starcitygames.com/catalog/category/Zendikar,,
http://www.starcitygames.com/catalog/category/Duel%20Decks%20Blessed%20vs%20Cursed,,

Answer 1

你正在以正确的方式做每一件事。可以帮助您保持链接有序的东西是priority 在请求中。在此处查看文档：https://docs.scrapy.org/en/latest/topics/request-response.html#scrapy.http.Request

priority (int) – 此请求的优先级（默认为 0）。调度程序使用优先级来定义处理请求的顺序。具有更高优先级值的请求将更早执行。允许使用负值以表示相对较低的优先级。

因此，在您的start_requests 中，您可以将priority 设置为例如等于从len(requests) 到1 的整数。并为parse 函数中的所有子项保留此优先级。

这将帮助您运行来自第一个 URL 的所有链接，其最高优先级等于 len(requests)，然后是来自第二个链接的所有链接，优先级为 len(requests) - 1 等等。