Python：从 http 到 csv答案

【问题标题】：Python: from http to csvPython：从 http 到 csv
【发布时间】：2018-06-08 10:52:41
【问题描述】：

我需要将HTTP 中的字符串写入 CSV 文件。我的专栏必须是：LATITUDE,LONGITUDE, OSM_ID, HIGHWAY, UPDATED_AT

这是从顶部开始的 HTTP 链接输出示例：

{
  "datetime": "2018-06-08T08:26:09.375Z",
  "success": true,
  "bbox": {
    "xmin": "12.335513",
    "ymin": "42.035682",
    "xmax": "12.758896",
    "ymax": "42.050826"
  },
  "data": [
    {
      "aggregate_id": 30201274,
      "ppe": 0.316954620298806,
      "geom": {
        "type": "Point",
        "coordinates": [
          12.532972800901,
          42.045435384225
        ]
      },
      "osm_id": "37015042",
      "highway": "motorway",
      "updated_at": "2018-01-20T03:27:11.047Z"
    },
    {
      "aggregate_id": 30201275,
      "ppe": 0.318124963244448,
      "geom": {
        "type": "Point",
        "coordinates": [
          12.5329908742,
          42.045615145535
        ]
      },
      "osm_id": "37015042",
      "highway": "motorway",
      "updated_at": "2018-01-20T03:27:11.047Z"
    },
    {
      "aggregate_id": 30201276,
      "ppe": 0.204792151096739,
      "geom": {
        "type": "Point",
        "coordinates": [
          12.533008947499,
          42.045794906844
        ]
      },
      "osm_id": "37015042",
      "highway": "motorway",
      "updated_at": "2018-01-20T03:27:11.047Z"
    },
    {
      "aggregate_id": 30201277,
      "ppe": 0.194797261691664,
      "geom": {
        "type": "Point",
        "coordinates": [
          12.533030586679,
          42.045974206816
        ]
      },
      "osm_id": "37015042",
      "highway": "motorway",
      "updated_at": "2018-01-20T03:27:11.047Z"
    }
  ]
}

每一行用','分隔。

我写了这段代码

import pandas as pd
import csv
import urllib.request

from urllib.request import urlopen

CSV_URL = 'http://www.smartroadsense.it/bb/12.335513/42.035682/12.758896/42.050826'

request = urllib.request.Request(CSV_URL)
response = urllib.request.urlopen(request)
response.read().decode('utf-8')

#write into csv
colNames = ["longitude","latitude","ppe","osm_id","highway","updated_at"]
data = pd.read_csv(CSV_URL, names=colNames, sep=',')

问题是如何将字符串从 http 拆分成行。有人可以帮助我吗？

【问题讨论】：

该问题与machine-learning 无关 - 请不要向标签发送垃圾邮件（已编辑和删除）。
它也与请求无关（标签已删除）。
您的源字符串格式不是 CSV，而是 JSON。使用json.load() 将其解析为Python dict，然后从该dict 构建你的csv（你不需要panda，stdlib 的csv 模块就足够了）。

标签： python json csv

【解决方案1】：

这是一个使用pandas 和requests 的简单实现：

import pandas as pd
import requests

url = 'http://www.smartroadsense.it/bb/12.335513/42.035682/12.758896/42.050826'
response = requests.get(url).json()

df = pd.DataFrame(response['data'])

您可以像这样从geom 列中提取longitude 和latitude：

df['longitude'] = df.apply(lambda row: dict(row['geom'])['coordinates'][0], axis=1)
df['latitude'] = df.apply(lambda row: dict(row['geom'])['coordinates'][1], axis=1)

最后，将所需的列保存到csv 文件中，如下所示：

df[['latitude', 'longitude', 'osm_id', 'highway', 'updated_at']].to_csv('output.csv', index=False)

【讨论】：

感谢您的宝贵时间！我还有一个问题：你能解释一下下面的说明吗？ 'lambda row: dict(row['geom'])['coordinates'][0]'，特别是我不明白程序如何读取每一行。
apply 函数（axis=1）对 DataFrame 的所有行应用一些转换。

【解决方案2】：

这是另一种方法：

import pandas as pd
import urllib.request
from urllib.request import urlopen
CSV_URL = "http://www.smartroadsense.it/bb/12.335513/42.035682/12.758896/42.050826"

request = urllib.request.Request(CSV_URL)
response = urllib.request.urlopen(request)
test = json.loads(response.read())["data"]

def transform_row(x):
  x["longitude"] = x["geom"]["coordinates"][0]
  x["latitude"] = x["geom"]["coordinates"][1]
  del x["geom"]
  return x

res = list(map(transform_row, test))
pd.DataFrame(res)[['latitude', 'longitude', 'osm_id', 'highway', 'updated_at']].to_csv('output.csv', index=False)

【讨论】：

【解决方案3】：

一个简单的方法。

获取数据
对数据排序
导出为 CSV

import csv
import requests


def main():
    url = 'http://www.smartroadsense.it/bb/12.335513/42.035682/12.758896/42.050826'
    data = fetch_data(url)
    rows = [row(r) for r in data]
    fields = ['LATITUDE', 'LONGITUDE', 'OSM_ID', 'HIGHWAY', 'UPDATED_AT']
    with open('output.csv', "w") as output:
        writer = csv.writer(output, lineterminator='\n')
        writer.writerows([fields])
        writer.writerows(rows)


def row(data):
    r = list()
    r.append(data['geom']['coordinates'][0])  # LATITUDE
    r.append(data['geom']['coordinates'][1])  # LONGITUDE
    r.append(data['osm_id'])  # OSM_ID
    r.append(data['highway'])  # HIGHWAY
    r.append(data['updated_at'])  # UPDATED_AT
    return r


def fetch_data(url):
    response = requests.get(url).json()
    return response['data']

【讨论】：

【解决方案4】：

除了其他答案之外，这是使用csv.writer 的另一种方法：

from requests import get
from csv import writer

CSV_URL = 'http://www.smartroadsense.it/bb/12.335513/42.035682/12.758896/42.050826'

response = get(CSV_URL).json()

headers = ['LATITUDE', 'LONGITUDE', 'OSM_ID', 'HIGHWAY', 'UPDATED_AT']
columns = ["osm_id", "highway", "updated_at"]

with open('output.csv', 'w') as out:
    csv_writer = writer(out)
    csv_writer.writerow(headers)
    for row in response['data']:
        items = row['geom']['coordinates'] + [row[col] for col in columns]
        csv_writer.writerow(items)

print(open('output.csv').read())

输出这个 csv 文件：

LATITUDE,LONGITUDE,OSM_ID,HIGHWAY,UPDATED_AT
12.532972800901,42.045435384225,37015042,motorway,2018-01-20T03:27:11.047Z
12.5329908742,42.045615145535,37015042,motorway,2018-01-20T03:27:11.047Z
12.533008947499,42.045794906844,37015042,motorway,2018-01-20T03:27:11.047Z
12.533030586679,42.045974206816,37015042,motorway,2018-01-20T03:27:11.047Z
...

【讨论】：