在scrapy中，我试图检索链接列表，然后从单个scrapy文件中的这些链接中抓取数据。我的 csv 文件返回一个 xpath 列表。

Question

我尝试在此页面上使用 igaggini 的示例，但似乎无法让它与我的代码一起使用。 Scrapy: Follow link to get additional Item data?

我很确定我有正确的 xpath，输出应该是从国家页面抓取的链接的第一个 div 中的第二段。

这是我的主文件，recursive.py。

from scrapy.spider import BaseSpider
from bathUni.items import BathuniItem
from scrapy.selector import HtmlXPathSelector
from scrapy.http.request import Request
from urlparse import urljoin

class recursiveSpider(BaseSpider):
name = 'recursive'
allowed_domains = ['http://www.bristol.ac.uk/']
start_urls = ['http://www.bristol.ac.uk/international/countries/']

def parse(self, response):
    hxs = HtmlXPathSelector(response)
    links = []

    #scrap main page to get row links
    for i in range(1, 154):
        xpath = ('//*[@id="all-countries"]/li[*]/ul/li[*]/a' .format (i+1))
        link = hxs.select(xpath).extract()
        links.append(link)

    #parse links to get content of the linked pages
    for link in links:
        item = BathuniItem()
        item ['Qualification'] = hxs.select('//*[@id="uobcms-content"]/div/div/div[1]/p[2]')

        yield item

这是我的物品文件

from scrapy.item import Item, Field

class BathuniItem(Item):
    Country = Field()
    Qualification = Field()

而且我收到的输出不是我想要的，我的 csv 文件充满了这些 -

<HtmlXPathSelector xpath='//*[@id="all-countries"]/li[*]/ul/li[*]/a' data=u'<a href="/international/countries/albani'>

Answer 1

您应该在选择器上调用.extract() 以获得有用的值，而不是SelectorList

item ['Qualification'] = hxs.select('//*[@id="uobcms-content"]/div/div/div[1]/p[2]').extract()

另外，我知道你想获取链接对应的页面

#parse links to get content of the linked pages
for link in links:
    item = BathuniItem()
    item ['Qualification'] = hxs.select('//*[@id="uobcms-content"]/div/div/div[1]/p[2]').extract()

    yield item

该代码不会获取这些链接页面，您需要产生额外的Requests 来告诉 Scrapy 下载它们。

你应该这样做：

    start_urls = ['http://www.bristol.ac.uk/international/countries/']

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        links = []

        #scrap main page to get row links
        for i in range(1, 154):
            xpath = ('//*[@id="all-countries"]/li[*]/ul/li[*]/a/@href' .format (i+1))
            links.extend(hxs.select(xpath).extract())

        #parse links to get content of the linked pages
        for link in links:
            yield Request(link, callback=self.parse_linkpage)

    def parse_linkpage(self, response):
        hxs = HtmlXPathSelector(response)
        item = BathuniItem()
        item ['Qualification'] = hxs.select('//*[@id="uobcms-content"]/div/div/div[1]/p[2]').extract()
        return item