将 json 编辑为 csv 输出文件答案

【问题标题】：Edit json to csv output files将 json 编辑为 csv 输出文件
【发布时间】：2021-04-26 18:34:39
【问题描述】：

我正在尝试通过'curl'获取JSON数据并使用'jq'命令将其转换为'csv'。

代码：

#!/bin/bash
 
    #Remove previously generated output files 
    rm datacsv.csv  output.json 
   #Get crypto pairs data from the server (curl) in JSON from the COINAMRKET.COM server, save output.json 
              curl -H "X-CMC_PRO_API_KEY: 45f65521-ad1b-4dbb-9c33-743ba7a63e68" -H "Accept: application/json" -d "start=1&limit=50&convert=USD" -G https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest>output.json 
                             
     #Convert JSON to CSV, Outputfile: datacsv.csv                               
      jq -r '.data[]| [paths(scalars) | select( all(.[]; type=="string")) ] as $paths| ($paths |  map(["data"] + . | join("__"))),[ getpath($paths[]) ]|@tsv'  output.json  >datacsv.csv

输出 csv 文件 (datacsv.csv) 的每条记录都有重复的标题：

data__id    data__name  data__symbol    data__slug  data__num_market_pairs  data__date_added    data__max_supply    data__circulating_supply    data__total_supply  data__cmc_rank  data__last_updated  data__quote__USD__price data__quote__USD__volume_24h    data__quote__USD__percent_change_1h data__quote__USD__percent_change_24h    data__quote__USD__percent_change_7d data__quote__USD__percent_change_30d    data__quote__USD__percent_change_60d    data__quote__USD__percent_change_90d    data__quote__USD__market_cap    data__quote__USD__last_updated                  
1   Bitcoin BTC bitcoin 9547    2013-04-28T00:00:00.000Z    21000000    18691931    18691931    1   2021-04-26T16:21:02.000Z    53285.1129054482    59530205038.6485    -0.6500853  5.53203621  -3.31916623 -3.33187743 4.07075949  69.03589533 996001653755.848    2021-04-26T16:21:02.000Z                    
data__id    data__name  data__symbol    data__slug  data__num_market_pairs  data__date_added    data__circulating_supply    data__total_supply  data__cmc_rank  data__last_updated  data__quote__USD__price data__quote__USD__volume_24h    data__quote__USD__percent_change_1h data__quote__USD__percent_change_24h    data__quote__USD__percent_change_7d data__quote__USD__percent_change_30d    data__quote__USD__percent_change_60d    data__quote__USD__percent_change_90d    data__quote__USD__market_cap    data__quote__USD__last_updated                      
1027    Ethereum    ETH ethereum    6312    2015-08-07T00:00:00.000Z    115629413.749   115629413.749   2   2021-04-26T16:21:02.000Z    2482.06593555852    36892666790.3224    -0.23795873 5.43647682  16.16445289 46.51867774 51.11877441 94.25117671 286999829014.995    2021-04-26T16:21:02.000Z                        
1839    Binance Coin    BNB binance-coin    579 2017-07-25T00:00:00.000Z    170532785   153432897   169432897   3   2021-04-26T16:20:10.000Z    533.23090641431 4373867739.9739 0.03999376  3.29006408  8.84917842  103.69391599    108.02152044    1217.45571414   81815162741.0835    2021-04-26T16:20:10.000Z                    
52  XRP XRP xrp 664 2013-08-04T00:00:00.000Z    100000000000    45404028640 99990831162 4   2021-04-26T16:21:03.000Z    1.24569103514058    13547433610.3615    0.9120132   9.79015802  -4.60461388 125.54110459    162.86485803    372.19710035    56559391436.1141    2021-04-26T16:21:03.000Z                    
data__id    data__name  data__symbol    data__slug  data__num_market_pairs  data__date_added    data__circulating_supply    data__total_supply  data__platform__id  data__platform__name    data__platform__symbol  data__platform__slug    data__platform__token_address   data__cmc_rank  data__last_updated  data__quote__USD__price data__quote__USD__volume_24h    data__quote__USD__percent_change_1h data__quote__USD__percent_change_24h    data__quote__USD__percent_change_7d data__quote__USD__percent_change_30d    data__quote__USD__percent_change_60d    data__quote__USD__percent_change_90d    data__quote__USD__market_cap    data__quote__USD__last_updated  
825 Tether  USDT    tether  12325   2015-02-25T00:00:00.000Z    50006254438.5217    51866290994.3298    1027    Ethereum    ETH ethereum    0xdac17f958d2ee523a2206206994597c13d831ec7  5   2021-04-26T16:20:10.000Z    0.99997577777133    117822325986.94 -0.01985404 0.00891016  0.00656415  -0.1262222  -0.05070907 -0.14745816 50005043175.5917    2021-04-26T16:20:10.000Z    
2010    Cardano ADA cardano 259 2017-10-01T00:00:00.000Z    45000000000 31948309440.7478    45000000000 6   2021-04-26T16:20:10.000Z    1.2163725547586 3494400505.85215    -0.63516459 6.44253956  1.77358151  2.5727442   8.69333792  266.42366354    38861046774.6607    2021-04-26T16:20:10.000Z                    
74  Dogecoin    DOGE    dogecoin    331 2013-12-15T00:00:00.000Z    129348760639.944    129348760639.944    7   2021-04-26T16:21:03.000Z    0.26639640673017    6963087140.98019    -0.74842096 -1.96582346 -25.83236676    387.41281461    386.8290754 3173.67302946   34458045049.482 2021-04-26T16:21:03.000Z                        
6636    Polkadot    DOT polkadot-new    186 2020-08-19T00:00:00.000Z    932996847.726976    1068520962.58116    8   2021-04-26T16:21:06.000Z    32.3141434419601    2146407823.73721    -0.76428061 3.64699209  -7.2706416  0.41872077  -5.75544689 96.74696139 30148993968.3461    2021-04-26T16:21:06.000Z                        
data__id    data__name  data__symbol    data__slug  data__num_market_pairs  data__date_added    data__max_supply    data__circulating_supply    data__total_supply  data__platform__id  data__platform__name    data__platform__symbol  data__platform__slug    data__platform__token_address   data__cmc_rank  data__last_updated  data__quote__USD__price data__quote__USD__volume_24h    data__quote__USD__percent_change_1h data__quote__USD__percent_change_24h    data__quote__USD__percent_change_7d data__quote__USD__percent_change_30d    data__quote__USD__percent_change_60d    data__quote__USD__percent_change_90d    data__quote__USD__market_cap    data__quote__USD__last_updated
7083    Uniswap UNI uniswap 235 2020-09-17T00:00:00.000Z    1000000000  523384243.630469    1000000000  1027    Ethereum    ETH ethereum    0x1f9840a85d5af5bf1d1762f925bdaddc4201f984  9   2021-04-26T16:20:10.000Z    35.2645393307245    1159198004.59251    -0.72129315 3.06863351  17.86724054 24.69707747 36.43669393 187.42707789    18456904244.5882    2021-04-26T16:20:10.000Z
2   Litecoin    LTC litecoin    741 2013-04-28T00:00:00.000Z    84000000    66752414.5153875    66752414.5153875    10  2021-04-26T16:21:02.000Z    241.772049644181    4873850130.55793    0.08569225  3.079198    -6.01020181 32.7958836  20.24742388 84.55115332 16138868076.0832    2021-04-26T16:21:02.000Z                    
1831    Bitcoin Cash    BCH bitcoin-cash    586 2017-07-23T00:00:00.000Z    21000000    18719506.25 18719506.25 11  2021-04-26T16:20:08.000Z    825.995177012163    4098896768.46408    -0.62457646 3.13607248  -8.10372907 65.63228051 51.71459366 96.05126285 15462221878.549 2021-04-26T16:20:08.000Z                    
1975    Chainlink   LINK    chainlink   495 2017-09-20T00:00:00.000Z    1000000000  419009556.434445    1000000000  1027    Ethereum    ETH ethereum    0x514910771af9ca656af840dff83e8264ecf986ca  12  2021-04-26T16:20:10.000Z    34.4665578984855    1711065668.01118    0.39416876  3.37393637  -5.31781365 29.84576669 22.2383717  55.53806504 14441817136.8665    2021-04-26T16:20:10.000Z
5426    Solana  SOL solana  41  2020-04-10T00:00:00.000Z    269856622.936333    493886983.051495    13  2021-04-26T16:21:05.000Z    46.722213592056 1743822879.60336    -1.09733007 3.84673072  46.93695628 195.9965083 184.71221177    1146.97970713   12608298776.0623    2021-04-26T16:21:05.000Z

我想过滤输出 'csv' 文件，获取感兴趣的字段 (data__id,data__name,data__symbol,data__slug,data__num_market_pairs,data__date_added,data__max_supply,data__circulating_supply,@ 987654336@)，不包括休息。

期望的输出：

data__id    data__name  data__symbol    data__slug  data__num_market_pairs  data__date_added    data__max_supply    data__circulating_supply    data__total_supply
1   Bitcoin BTC bitcoin 9547    2013-04-28T00:00:00.000Z    21000000    18691931    18691931
1027    Ethereum    ETH ethereum    6312    2015-08-07T00:00:00.000Z    115629413.749   115629413.749   2
1839    Binance Coin    BNB binance-coin    579 2017-07-25T00:00:00.000Z    170532785   153432897   169432897
52  XRP XRP xrp 664 2013-08-04T00:00:00.000Z    100000000000    45404028640 99990831162
825 Tether  USDT    tether  12325   2015-02-25T00:00:00.000Z    50006254438.5217    51866290994.3298    1027
2010    Cardano ADA cardano 259 2017-10-01T00:00:00.000Z    45000000000 31948309440.7478    45000000000
74  Dogecoin    DOGE    dogecoin    331 2013-12-15T00:00:00.000Z    129348760639.944    129348760639.944    7
6636    Polkadot    DOT polkadot-new    186 2020-08-19T00:00:00.000Z    932996847.726976    1068520962.58116    8
7083    Uniswap UNI uniswap 235 2020-09-17T00:00:00.000Z    1000000000  523384243.630469    1000000000
2   Litecoin    LTC litecoin    741 2013-04-28T00:00:00.000Z    84000000    66752414.5153875    66752414.5153875
1831    Bitcoin Cash    BCH bitcoin-cash    586 2017-07-23T00:00:00.000Z    21000000    18719506.25 18719506.25
1975    Chainlink   LINK    chainlink   495 2017-09-20T00:00:00.000Z    1000000000  419009556.434445    1000000000
5426    Solana  SOL solana  41  2020-04-10T00:00:00.000Z    269856622.936333    493886983.051495    13

【问题讨论】：

标签： json bash csv jq

【解决方案1】：

要只发出一次标头，您可以使用模板：HEADERS, TABLE:

def relevantPaths:
   "id,name,symbol,slug,num_market_pairs,date_added,max_supply,circulating_supply,total_supply"
  | split(",")
  | map([.]) ;

relevantPaths as $paths
| ([["data"] + $paths[]] | map(join("__"))),
  (.data[] | [ getpath($paths[]) ])
| @tsv

【讨论】：

这适用于以“data__”开头的字段，例如：“id,cmc_rank,name,symbol,slug,num_market_pairs,max_supply” 但它会留下以“data__platform”和“data__quote__USD”开头的空字段例如：data__platform__id、data__platform__name、data__platform__symbol、data__platform__slug 和 data__quote__USD__percent_change_90d、data__quote__USD__market_cap、data__quote__USD__last_updated
@mehdishokati - SO 是一个问答网站，而不是免费的 SE 服务。如果您的 Q 已被回答，那么如果您接受它并从那里继续前进将会很有帮助。在这种情况下，“继续前进”可能意味着调整已提供的解决方案，或与 SE 合作为您这样做，或根据minimal reproducible example 指南在 SO 上提出新的 Q。
非常感谢您的回答和cmets，我将在SO中添加一个新Q。