如何让scrapy spider根据CSV文件向项目添加信息

Question

正如你们中的一些人可能已经收集到的那样，我正在学习 scrapy 以从 Google Scholar 中抓取一些数据，用于我正在运行的一个研究项目。我有一个文件，其中包含我正在抓取引用的许多文章标题。我使用 pandas 读取文件，生成需要抓取的 URL，然后开始抓取。

我面临的一个问题是 503 错误。谷歌很快就把我关闭了，许多条目仍然没有被刮掉。这是我正在使用 Crawlera 提供的一些中间件解决的问题。

我面临的另一个问题是，当我导出抓取的数据时，我很难将抓取的数据与我想要查找的数据相匹配。我的输入数据是一个包含三个字段的 CSV 文件——“作者”、“标题”、“pid”，其中“pid”是唯一标识符。

我使用 pandas 读取文件并根据标题为学者生成 URL。每次抓取给定的 URL 时，我的蜘蛛程序都会浏览该学术网页，并为该页面上列出的每篇文章获取标题、出版信息和引用。

这是我如何生成用于抓取的链接：

class ScholarSpider(Spider):
    name = "scholarscrape"
    allowed_domains = ["scholar.google.com"]

    # get the data
    data = read_csv("../../data/master_jeea.csv")
    # get the titles
    queries = data.Title.apply(urllib.quote)
    # generate a var to store links
    links = []
    # create the URLs to crawl
    for entry in queries:
        links.append("http://scholar.google.com/scholar?q=allintitle%3A"+entry)
    # give the URLs to scrapy
    start_urls = links

例如，我的数据文件中的一个标题可能是 Rodney Brooks 的论文 'Elephants Don't Play Chess'，'pid' 5067。蜘蛛会去

http://scholar.google.com/scholar?q=allintitle%3Aelephants+don%27t+play+chess

现在在这个页面上，有六个点击。蜘蛛获得了所有六次命中，但需要为它们分配相同的“pid”。我知道我需要在某处插入一行，内容类似于 item['pid'] = data.pid.apply("something") 但我不知道该怎么做。

下面是我的蜘蛛的其余代码。我确信这样做的方法非常简单，但我想不出如何让蜘蛛知道如果有意义的话它应该寻找哪个 data.pid 条目。

def parse(self, response):
    # initialize something to hold the data
    items=[]
    sel = Selector(response)
    # get each 'entry' on the page
    # an entry is a self contained div
    # that has the title, publication info
    # and cites
    entries = sel.xpath('//div[@class="gs_ri"]')
    # a counter for the entry that is being scraped
    count = 1
    for entry in entries:
        item = ScholarscrapeItem()
        # get the title
        title = entry.xpath('.//h3[@class="gs_rt"]/a//text()').extract()
        # the title is messy
        # clean up
        item['title'] = "".join(title)
        # get publication info
        # clean up
        author = entry.xpath('.//div[@class="gs_a"]//text()').extract()
        item['authors'] = "".join(author)
        # get the portion that contains citations
        cite_string = entry.xpath('.//div[@class="gs_fl"]//text()').extract()
        # find the part that says "Cited by"
        match = re.search("Cited by \d+",str(cite_string))
        # if it exists, note the number
        if match:
            cites = re.search("\d+",match.group()).group()
        # if not, there is no citation info
        else:
            cites = None
        item['cites'] = cites
        item['entry'] = count
        # iterate the counter
        count += 1
        # append this item to the list
        items.append(item)
    return items

我希望这个问题是明确定义的，但如果我能更清楚，请告诉我。除了顶部的一些导入内容之外，我的爬虫中真的没有太多其他内容。

编辑 1：根据以下建议，我将代码修改如下：

# test-case: http://scholar.google.com/scholar?q=intitle%3Amigratory+birds
import re
from pandas import *
import urllib

from scrapy.spider import Spider
from scrapy.selector import Selector

from scholarscrape.items import ScholarscrapeItem

class ScholarSpider(Spider):
    name = "scholarscrape"
    allowed_domains = ["scholar.google.com"]

    # get the data
    data = read_csv("../../data/master_jeea.csv")
    # get the titles
    queries = data.Title.apply(urllib.quote)
    pid = data.pid
    # generate a var to store links
    urls = []
    # create the URLs to crawl
    for entry in queries:
        urls.append("http://scholar.google.com/scholar?q=allintitle%3A"+entry)
    # give the URLs to scrapy
    start_urls = (
        (urls, pid),
        )

    def make_requests_from_url(self, (url,pid)):
        return Request(url, meta={'pid':pid}, callback=self.parse, dont_filter=True)

    def parse(self, response):
        # initialize something to hold the data
        items=[]
        sel = Selector(response)
        # get each 'entry' on the page
        # an entry is a self contained div
        # that has the title, publication info
        # and cites
        entries = sel.xpath('//div[@class="gs_ri"]')
        # a counter for the entry that is being scraped
        count = 1
        for entry in entries:
            item = ScholarscrapeItem()
            # get the title
            title = entry.xpath('.//h3[@class="gs_rt"]/a//text()').extract()
            # the title is messy
            # clean up
            item['title'] = "".join(title)
            # get publication info
            # clean up
            author = entry.xpath('.//div[@class="gs_a"]//text()').extract()
            item['authors'] = "".join(author)
            # get the portion that contains citations
            cite_string = entry.xpath('.//div[@class="gs_fl"]//text()').extract()
            # find the part that says "Cited by"
            match = re.search("Cited by \d+",str(cite_string))
            # if it exists, note the number
            if match:
                cites = re.search("\d+",match.group()).group()
            # if not, there is no citation info
            else:
                cites = None
            item['cites'] = cites
            item['entry'] = count
            item['pid'] = response.meta['pid']
            # iterate the counter
            count += 1
            # append this item to the list
            items.append(item)
        return items

Answer 1

您需要使用元组 (url, pid) 填充您的列表 start_urls。 现在重新定义方法make_requests_from_url(url)：

class ScholarSpider(Spider):
    name = "ScholarSpider"
    allowed_domains = ["scholar.google.com"]
    start_urls = (
        ('http://www.scholar.google.com/', 100),
        )

    def make_requests_from_url(self, (url, pid)):
        return Request(url, meta={'pid': pid}, callback=self.parse, dont_filter=True)

    def parse(self, response):
        pid = response.meta['pid']
        print '!!!!!!!!!!!', pid, '!!!!!!!!!!!!'
        pass