请求 .gz 文件，然后作为数据框打开

Question

这里是非常新的 python 用户。我有一个压缩数据集 (csv.gz) 的链接，我试图在没有先下载到我的桌​​面、解压缩并上传到 github 以使用 pd.read_csv 的情况下进入 python。该链接的形式是我认为我应该使用请求来请求它？但我得到一个 403 状态码。我不知道是因为我请求错误，还是 .gz 是问题？

这就是我尝试导入它的方式。

import requests
query = {'AWSAccessKeyId':'theaccesskey', 'Signature':'thesignature','Expires':'12345'}
url = 'https://s3.amazonaws.com/research.insideairbnb.com/data/united-states/or/portland/2015-12-02/data/listings.csv.gz'

response = requests.get(url, params=query) 

但即使假设我可以弄清楚如何获得 200 状态码，我如何从 .gz 转到我可以使用 pandas 的数据框？

Answer 1

您可以将io.BytesIO 与pandas.read_csv 一起使用：

import requests
import pandas as pd
from io import BytesIO

url = 'https://github.com/apache-superset/examples-data/blob/master/san_francisco.csv.gz'

# This is required to access raw binary files on Github
# i.e. it appends the following to the URL: `?raw=true`.
query = {'raw': 'true'} 

# Use the requests module to parse the URL with the provided parameters.
response = requests.get(url, params=query)

# Create a file pointer initialized to the content of the response
# using `BytesIO`. This is a psuedo-file, which can now be read
# using `pandas.read_csv`. Since `response.content` is binary data
# i.e. bytes, we use `BytesIO`. If the response was text, we would
# have used `StringIO`.
fp = BytesIO(response.content)

# Finally, parse the content into a DataFrame 
# (populate other parameters as needed).
df = pd.read_csv(fp, compression='gzip')
print(df)

这应该将csv.gz 文件的内容作为DataFrame 返回。在此示例中使用 URL 会产生以下输出：

               LON        LAT NUMBER            STREET UNIT  CITY  DISTRICT  REGION  POSTCODE  ID
0      -122.391267  37.769093   1550       04th Street  NaN   NaN       NaN     NaN     94158 NaN
1      -122.390850  37.769426   1505       04th Street  NaN   NaN       NaN     NaN     94158 NaN
2      -122.428577  37.780627   1160   Buchanan Street  NaN   NaN       NaN     NaN     94115 NaN
3      -122.428534  37.780385   1142   Buchanan Street  NaN   NaN       NaN     NaN     94115 NaN
4      -122.428525  37.780317   1140   Buchanan Street  NaN   NaN       NaN     NaN     94115 NaN
...            ...        ...    ...               ...  ...   ...       ...     ...       ...  ..
261547 -122.418380  37.808349    360  Jefferson Street  NaN   NaN       NaN     NaN     94133 NaN
261548 -122.418380  37.808349    350  Jefferson Street  NaN   NaN       NaN     NaN     94133 NaN
261549 -122.417829  37.807479    333  Jefferson Street  NaN   NaN       NaN     NaN     94133 NaN
261550 -122.418916  37.809044   1965      Al Scoma Way  NaN   NaN       NaN     NaN     94133 NaN
261551 -122.444322  37.749124    350    Glenview Drive  NaN   NaN       NaN     NaN     94131 NaN

我使用了我在网上找到的示例 csv.gz 文件，因为 URL 和参数会生成 JPEG 图像，这有点令人费解。无论如何，根据您的情况调整这段代码，它应该会产生所需的结果。