我正在尝试使用 Scrapy 抓取数据

Question

我正在尝试从 pewdiepie 频道获取所有视频链接。我写了以下代码，它没有显示任何错误，但它没有抓取链接。

代码如下：

import scrapy
from scrapy.crawler import CrawlerProcess

class PewSpider(scrapy.Spider):
   name = "pew_spider"
   def start_request(self):
      urls = ['https://www.youtube.com/user/PewDiePie/videos'] 
      for url in urls:
        yield scrapy.Request(url=url, callback=self.parser)

   def parser(self, response):
      links = response.css('div#contents > a#thumbnail::attr(href)')
      filepath = "./Desktop/pew.csv"
      with open(filepath, 'w') as f:
         f.writelines( [link + '/n' for link in links])

process = CrawlerProcess()
process.crawl(PewSpider)
process.start()

Answer 1

我想你应该先看看 YouTube API，然后再尝试从网站上抓取它。 https://developers.google.com/youtube/v3