【发布时间】：2019-03-11 15:00:55
【问题描述】：
我正在尝试使用 scrapy 来抓取数据。我已经根据需要获得了文本数据。但是当我尝试废弃图像 src 时，它会在开始时返回我确切的 url，并在一些记录后返回
"data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=="
这是我的蜘蛛代码
import scrapy
class CoinmarketcapSpider(scrapy.Spider):
name = 'coinmarketcap'
allowed_domains = ['coinmarketcap.com']
start_urls = ['https://coinmarketcap.com/2']
def parse(self, response):
cointable=response.css('table#currencies').xpath('//tbody/tr')
for coins in cointable:
name=coins.css('a.currency-name-container::text').extract_first().strip()
logo=coins.css('img.logo-sprite::attr(src)').extract()
symbol=coins.css('span.currency-symbol').xpath('.//a/text()').extract_first().strip()
market_cap=coins.css('.market-cap').xpath('text()').extract_first().strip()
yield {
'Name':name,
'image_urls':logo,
'symbol':symbol,
'market_cap':market_cap
}
print response
这里有 ImgLogo url 的输出 json 文件
[ {"coinName": ["Bitcoin"], "symbol": ["BTC"], "imgLogo": ["https://s2.coinmarketcap.com/static/img/coins/16x16/1.png"]}, {“coinName”：[“以太坊”]，“符号”：[“ETH”]，“imgLogo”： ["https://s2.coinmarketcap.com/static/img/coins/16x16/1027.png"]}, {“coinName”：[“XRP”]，“符号”：[“XRP”]，“imgLogo”： ["https://s2.coinmarketcap.com/static/img/coins/16x16/52.png"]}, {“coinName”：[“比特币现金”]，“符号”：[“BCH”]，“imgLogo”： ["https://s2.coinmarketcap.com/static/img/coins/16x16/1831.png"]}, {“coinName”：[“EOS”]，“符号”：[“EOS”]，“imgLogo”： ["https://s2.coinmarketcap.com/static/img/coins/16x16/1765.png"]}, {“coinName”：[“Stellar”]，“符号”：[“XLM”]，“imgLogo”： ["https://s2.coinmarketcap.com/static/img/coins/16x16/512.png"]}, {"coinName": ["Litecoin"], "symbol": ["LTC"], "imgLogo": ["https://s2.coinmarketcap.com/static/img/coins/16x16/2.png"]}, {"coinName": ["Tether"], "symbol": ["USDT"], "imgLogo": ["https://s2.coinmarketcap.com/static/img/coins/16x16/825.png"]}, {“coinName”：[“Cardano”]，“符号”：[“ADA”]，“imgLogo”： ["https://s2.coinmarketcap.com/static/img/coins/16x16/2010.png"]}, {“coinName”：[“Monero”]，“符号”：[“XMR”]，“imgLogo”： ["https://s2.coinmarketcap.com/static/img/coins/16x16/328.png"]}, {“coinName”：[“IOTA”]，“符号”：[“MIOTA”]，“imgLogo”： ["数据:图像/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAAAAAAAAAAICTAEAOw=="]}, {“coinName”：[“TRON”]，“符号”：[“TRX”]，“imgLogo”： ["数据:图像/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAAAAAAAAAAICTAEAOw=="]}, {"coinName": ["Dash"], "symbol": ["DASH"], "imgLogo": ["data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=="]}]
这是我的 Items.py 代码
import scrapy
class CmindexItem(scrapy.Item):
# define the fields for your item here like:
image_urls = scrapy.Field()
这是我的 piplines.py 代码，用于下载图像并将它们保存在我的图像目录中
import scrapy
from scrapy.pipelines.images import ImagesPipeline
from scrapy.exceptions import DropItem
class CmindexPipeline(ImagesPipeline):
def get_media_requests(self, item, info):
for image_url in item['image_urls']:
yield scrapy.Request(image_url)
def item_completed(self, results, item, info):
image_paths = [x['path'] for ok, x in results if ok]
if not image_paths:
raise DropItem("Item contains no images")
item['image_paths'] = image_paths
return item
这是我的 settings.py 代码
BOT_NAME = 'cmindex'
SPIDER_MODULES = ['cmindex.spiders']
NEWSPIDER_MODULE = 'cmindex.spiders'
USER_AGENT = 'cmindex (+http://www.cmindex.com)'
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
ITEM_PIPELINES = {'cmindex.pipelines.CmindexPipeline': 1}
IMAGES_STORE ='E:\WorkPlace\cmindex\cmindex\img'
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}
我添加了假用户代理，并且还添加了请求延迟，但这并没有影响我的最终结果。所以如果有人有想法请分享。谢谢
【问题讨论】：
标签： python-2.7 xpath scrapy