抓取图像并保存 jpg

Question

我在抓取图像时遇到了一些问题。

Settings.py：

ITEM_PIPELINES = {
   'scrapy.contrib.pipeline.images.FilesPipeline': 1,
}
FILES_STORE = 'D:/0. Documentos/10. GitHub/0. 93Pipe/RealEstatePredictor/Images'
Items.py

导入scrapy

class MagazineCover(scrapy.Item):
    title = scrapy.Field()
    pubDate = scrapy.Field()
    file_urls = scrapy.Field()
    files = scrapy.Field()
Scrapy spider:

imgs = response.css('div.gallery-content.item-gallery__wrapper  img::attr(src)').getall()
    

    for img in imgs:
        yield MagazineCover(title="title", pubDate="02/03/2021", file_urls=[img])

蜘蛛读取了网址，但是我在Settings.py中设置的文件夹是空的

结果

2021-02-04 15:24:17 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.portalinmobiliario.com/venta/departamento/conchali-metropolitana/6055529-julio-montt-salamanca-5935-uda>
{'file_urls': ['https://http2.mlstatic.com/julio-montt-salamanca-5935-D_NQ_NP_2X_760548-MLC44703096172_012021-R.jpg'],
 'pubDate': '02/03/2021',
 'title': 'title'}

感谢您的帮助！！

Answer 1

来自documentation：

对于图像管道，使用：

ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}

IMAGES_STORE = '/path/to/valid/dir'

对于文件管道，使用：

ITEM_PIPELINES = {'scrapy.pipelines.files.FilesPipeline': 1}

FILES_STORE = '/path/to/valid/dir'

没有scrapy.contrib.pipeline.images.FilesPipeline。

Answer 2

按照documentation保存图片需要三处修改

1.For Images Pipeline, use:

    ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}

2.For the Images Pipeline, set the IMAGES_STORE setting:
    IMAGES_STORE = '/path/to/valid/dir'

3. In items.py 

class MagazineCover(scrapy.Item):
      image_urls = scrapy.Field()

你的蜘蛛的结尾会改变这个

for img in imgs:
        yield MagazineCover(image_urls=[img])

希望对你有所帮助。