Webscraping Coinmarketcap [重复]答案

【问题标题】：Webscraping Coinmarketcap [duplicate]Webscraping Coinmarketcap [重复]
【发布时间】：2021-11-08 08:22:58
【问题描述】：

由于 coinmarketcap api 计划的历史数据受到限制，我正在寻求网络抓取。

然而，尽管阅读了关于属性的蹩脚文档，但我还是被困在了第一个障碍上。

import json 
import requests 
from bs4 import BeautifulSoup


r = requests.get('https://coinmarketcap.com/historical/20210905/')
soup = BeautifulSoup(r.text, 'lxml')
print(soup)

输出中包含我要抓取的数据。我要获取的数据：

2021 年 9 月 5 日 BTC 的市值、价格和流通供应量。

数据在<script id="__NEXT_DATA__" type="application/json"> 之后不久出现在输出中，因此我认为使用__NEXT_DATA__ 作为属性id 可以让我访问数据。不幸的是没有。

包含数据的数据结构示例如下：

"listingHistorical":{"data":[{"id":1,"name":"Bitcoin","symbol":"BTC","slug":"bitcoin","num_market_pairs":8848,"date_added":"2013-04-28T00:00:00.000Z","tags":["mineable","pow","sha-256","store-of-value","state-channels","coinbase-ventures-portfolio","three-arrows-capital-portfolio","polychain-capital-portfolio","binance-labs-portfolio","arrington-xrp-capital","blockchain-capital-portfolio","boostvc-portfolio","cms-holdings-portfolio","dcg-portfolio","dragonfly-capital-portfolio","electric-capital-portfolio","fabric-ventures-portfolio","framework-ventures","galaxy-digital-portfolio","huobi-capital","alameda-research-portfolio","a16z-portfolio","1confirmation-portfolio","winklevoss-capital","usv-portfolio","placeholder-ventures-portfolio","pantera-capital-portfolio","multicoin-capital-portfolio","paradigm-xzy-screener"],"max_supply":21000000,"circulating_supply":18807550,"total_supply":18807550,"platform":null,"cmc_rank":1,"last_updated":"2021-09-05T23:00:00.000Z","quote":{"BTC":{"price":1,"volume_24h":585906.8067215424,"percent_change_1h":0,"percent_change_24h":0,"percent_change_7d":0,"market_cap":18807550,"fully_diluted_market_cap":null,"last_updated":"2021-09-05T23:59:03.000Z"},"USD":{"price":51753.41192620951,"volume_24h":30322676318.63,"percent_change_1h":-0.159917099159,"percent_change_24h":3.621580803777,"percent_change_7d":5.987281074996,"market_cap":973354882472.7817,"last_updated":"2021-09-05T23:00:00.000Z"}},"rank":1,"noLazyLoad":true},

有没有简单的解决方案？

【问题讨论】：

“coinmarketcap api 计划的历史数据限制” 你确定吗？ Pricing 页面详细说明了历史数据可通过其 API 获得。
有点切题，但您应该知道目标站点的Terms of Service 明确禁止您尝试做的事情（“使用或向服务引入任何数据挖掘，抓取、“抓取”、机器人或类似的自动或数据收集或提取方法，” - 一旦提供商检测到此活动，您的访问可能会被完全阻止，从而破坏您可能构建的任何工作流程或应用程序此提取的顶部。
重复How to extract json from script tag using beautiful soup python?
感谢您的提醒。我没有意识到我会违反目标网站的服务条款。我现在找到了使用 coingecko API 的替代解决方案

标签： python json web-scraping beautifulsoup

【解决方案1】：

这仅适用于列表，它已完全加载到页面上。

https://coinmarketcap.com/historical/20210905/ -> 20210905 -> 2021-09-05 是日期，只需替换为所需的日期，它将显示数据https://coinmarketcap.com/historical/20210101/，然后抓取并提取JSON数据。

【讨论】：

【解决方案2】：

你可以试试这样的：

r = requests.get('https://coinmarketcap.com/historical/20210905/')
soup = BeautifulSoup(r.text)

data = json.loads(soup.find('script', type='application/ld+json', id='__NEXT_DATA__').text)

historical_data = data['listingHistorical']['data']
print historical_data

【讨论】：