scrapy 从一个简单的站点中提取 url

Question

我正在尝试从基本网站 vapedonia.com 提取基本数据。这是一个简单的电子商务网站，我很容易“重新发明轮子”（主要是在一个大的 html 字符串上工作），但是当我不得不在那个叫做 scrapy 的模具中工作时，它就不起作用了。

我首先分析 html 代码并使用插件创建我的 xpath 表达式。在那个插件中，一切都很好，但是当我创建我的代码时（甚至当我使用 scrappy shell 时），它不起作用。

代码如下：

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector

class MySpider(BaseSpider):
   name = "vapedonia"
   allowed_domains = ["vapedonia.com"]
   start_urls = ["https://www.vapedonia.com/23-e-liquidos"]

   def parse(self, response):
        hxs = HtmlXPathSelector(response)
        products = hxs.select("//div[@class='product-container clearfix']")
        for products in products:
            image = products.select("div[@class='center_block']/a/img/@src").extract()
            name = products.select("div[@class='center_block']/a/@title").extract()
            link = products.select("div[@class='right_block']/p[@class='s_title_block']/a/@href").extract()
            price = products.select("div[@class='right_block']/div[@class='content_price']/span[@class='price']").extract()
        print image, name, link, price

以下是错误：

C:\Users\eric\Documents\Web Scraping\0 - Projets\Scrapy-\projects\craigslist_sample>scrapy crawl vapedonia
C:\Users\eric\Documents\Web Scraping\0 - Projets\Scrapy-\projects\craigslist_sample\craigslist_sample\spiders\test.py:1: ScrapyDeprecationWarning: Module `scrapy.spider` is deprecated, use `scrapy.spiders` instead
  from scrapy.spider import BaseSpider
C:\Users\eric\Documents\Web Scraping\0 - Projets\Scrapy-\projects\craigslist_sample\craigslist_sample\spiders\test.py:6: ScrapyDeprecationWarning: craigslist_sample.spiders.test.MySpider inherits from deprecated class scrapy.spiders.BaseSpider, please inherit from scrapy.spiders.Spider. (warning only on first subclass, there may be others)
  class MySpider(BaseSpider):
C:\Users\eric\Documents\Web Scraping\0 - Projets\Scrapy-\projects\craigslist_sample\craigslist_sample\spiders\test2.py:1: ScrapyDeprecationWarning: Module `scrapy.contrib.spiders` is deprecated, use `scrapy.spiders` instead
  from scrapy.contrib.spiders import CrawlSpider, Rule
C:\Users\eric\Documents\Web Scraping\0 - Projets\Scrapy-\projects\craigslist_sample\craigslist_sample\spiders\test2.py:2: ScrapyDeprecationWarning: Module `scrapy.contrib.linkextractors` is deprecated, use `scrapy.linkextractors` instead
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
C:\Users\eric\Documents\Web Scraping\0 - Projets\Scrapy-\projects\craigslist_sample\craigslist_sample\spiders\test2.py:2: ScrapyDeprecationWarning: Module `scrapy.contrib.linkextractors.sgml` is deprecated, use `scrapy.linkextractors.sgml` instead
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
C:\Users\eric\Documents\Web Scraping\0 - Projets\Scrapy-\projects\craigslist_sample\craigslist_sample\spiders\test2.py:13: ScrapyDeprecationWarning: SgmlLinkExtractor is deprecated and will be removed in future releases. Please use scrapy.linkextractors.LinkExtractor
  Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_items", follow= True),
C:\Users\eric\Documents\Web Scraping\0 - Projets\Scrapy-\projects\craigslist_sample\craigslist_sample\spiders\test4.py:15: ScrapyDeprecationWarning: SgmlLinkExtractor is deprecated and will be removed in future releases. Please use scrapy.linkextractors.LinkExtractor
  Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_items", follow= True),
Traceback (most recent call last):
  File "C:\Users\eric\Miniconda2\Scripts\scrapy-script.py", line 5, in <module>
    sys.exit(scrapy.cmdline.execute())
  File "C:\Users\eric\Miniconda2\lib\site-packages\scrapy\cmdline.py", line 148, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "C:\Users\eric\Miniconda2\lib\site-packages\scrapy\crawler.py", line 243, in __init__
    super(CrawlerProcess, self).__init__(settings)
  File "C:\Users\eric\Miniconda2\lib\site-packages\scrapy\crawler.py", line 134, in __init__
    self.spider_loader = _get_spider_loader(settings)
  File "C:\Users\eric\Miniconda2\lib\site-packages\scrapy\crawler.py", line 330, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "C:\Users\eric\Miniconda2\lib\site-packages\scrapy\spiderloader.py", line 61, in from_settings
    return cls(settings)
  File "C:\Users\eric\Miniconda2\lib\site-packages\scrapy\spiderloader.py", line 25, in __init__
    self._load_all_spiders()
  File "C:\Users\eric\Miniconda2\lib\site-packages\scrapy\spiderloader.py", line 47, in _load_all_spiders
    for module in walk_modules(name):
  File "C:\Users\eric\Miniconda2\lib\site-packages\scrapy\utils\misc.py", line 71, in walk_modules
    submod = import_module(fullpath)
  File "C:\Users\eric\Miniconda2\lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
  File "C:\Users\eric\Documents\Web Scraping\0 - Projets\Scrapy-\projects\craigslist_sample\craigslist_sample\spiders\test5.py", line 17
    link = products.select("div[@class='right_block']/p[@class='s_title_block']/a/@href").extract()
    ^
IndentationError: unexpected indent

C:\Users\eric\Documents\Web Scraping\0 - Projets\Scrapy-\projects\craigslist_sample>

我不知道问题是什么，但我在蜘蛛目录/文件夹中有几个蜘蛛编码的蜘蛛。可能是蜘蛛之间的某种代码混合。

谢谢。

Answer 1

当 scrapy 运行时，它会扫描项目中存在的所有爬虫，以优化它们的名称并运行您指定的爬虫。因此，如果任何刮板有语法错误，那么它将无法工作

  File "C:\Users\eric\Documents\Web Scraping\0 - Projets\Scrapy-\projects\craigslist_sample\craigslist_sample\spiders\test5.py", line 17
    link = products.select("div[@class='right_block']/p[@class='s_title_block']/a/@href").extract()

正如您在异常中看到的那样，您的test5.py 中存在错误。修复此文件中的缩进或在不需要时对其进行注释。这应该允许你运行蜘蛛

Edit-1：混合制表符和空格

Python 依赖于缩进，看起来相同的缩进在代码中可能会有所不同。它可能在不同的行中混合使用制表符和空格。这将导致错误。因此，请务必检查您的编辑器以显示制表符和空格字符并将所有制表符转换为空格。