使用 numpy 或 pandas 处理长格式的 csv 文件答案

【问题标题】：Manipulating csv files from long format using numpy or pandas使用 numpy 或 pandas 处理长格式的 csv 文件
【发布时间】：2013-12-15 04:27:06
【问题描述】：

我正在尝试编写一个简单的脚本，将 csv 输出文件从 Fortran 代码转换为 Pandas DataFrame 对象，以便进行更多分析。 csv 有两列，但由多个附加的数据块组成，形状为 [n,2]（每个样本名称的格式为 RN_x）。我得到了以下代码，但生成的 DataFrame 对象不允许分析。我还在下面附上了一个示例文件（与原始文件相比缩短了很多）。顺便说一句，数据文件中的第一列是一个日期，但在输出中是一个数字，对应于 si=imulation 中的一天。任何建议将不胜感激。

import numpy as np
import pandas as pd
import csv as csv
readdata = csv.reader(open('C:/data/Test.csv', 'r'))
data = []
for row in readdata:
    data.append(row)
a = np.array(data).reshape(11,-1, order = 'F')
col = a[0,:4].reshape(4)
row = pd.Index(a[4:,0:1].reshape(7))
b = a[4:,5:]
df = pd.DataFrame(b, index = row, columns = col)

示例：

RN_48865,
1,Observed
1,0
259,Computed
1,0.000014
91,0.000014
182,0.000014
274,0.000014
366,0.000014
457,0.000014
548,0.000014
RN_7445,
1,Observed
1,0
259,Computed
1,0.000013
91,0.000013
182,0.000013
274,0.000013
366,0.000013
457,0.000013
548,0.000013
RN_9288,
1,Observed
1,0
259,Computed
1,0.000011
91,0.000011
182,0.000011
274,0.000011
366,0.000011
457,0.000011
548,0.000011
RN_10955,
1,Observed
1,0
259,Computed
1,0.000014
91,0.000014
182,0.000014
274,0.000014
366,0.000014
457,0.000014
548,0.000014

样本输出：

Index,RN_48865,RN_7445,RN_9288,RN_10955
1,0.000014,0.000013,0.000011,0.000014
91,0.000014,0.000013,0.000011,0.000014
182,0.000014,0.000013,0.000011,0.000014
274,0.000014,0.000013,0.000011,0.000014
366,0.000014,0.000013,0.000011,0.000014
457,0.000014,0.000013,0.000011,0.000014
548,0.000014,0.000013,0.000011,0.000014

【问题讨论】：

那么问题是什么？
抱歉，不清楚。如何将长文件转换为带有索引的 Dataframe 对象（将数字添加到基准日期的已解析日期，例如 1995-1-1；第一个数据列），以及用第二列中的数据填充的多列“RN_x”标签作为列标签。原始长文件具有重复的数据块，表示总和中不同“位置”的输出。我希望能够分析每个位置的统计信息。
我不明白“用“RN_x”标签作为列标签的第二列中的数据填充的多列。”为什么不简单地显示数据（使用\ns）？
我可以通过电子邮件将文件发送给您吗？
如果您向我们展示所需的输出以及包括空白字符在内的确切输入，也许会更清楚。

标签： python csv file-io numpy pandas

【解决方案1】：

您实际上是在问几个问题。这是我可以从所需的输出中理解的：

source="""RN_48865,
    1,Observed
    1,0
    259,Computed
    1,0.000014
    91,0.000014
    182,0.000014
    274,0.000014
    366,0.000014
    457,0.000014
    548,0.000014
    RN_7445,
    1,Observed
    1,0
    259,Computed
    1,0.000013
    91,0.000013
    182,0.000013
    274,0.000013
    366,0.000013
    457,0.000013
    548,0.000013
    RN_9288,
    1,Observed
    1,0
    259,Computed
    1,0.000011
    91,0.000011
    182,0.000011
    274,0.000011
    366,0.000011
    457,0.000011
    548,0.000011
    RN_10955,
    1,Observed
    1,0
    259,Computed
    1,0.000014
    91,0.000014
    182,0.000014
    274,0.000014
    366,0.000014
    457,0.000014
    548,0.000014
"""
import pandas as pd
import numpy as np
import StringIO
df = pd.read_csv(StringIO.StringIO(source), header=None)
rns = np.where(df[0].apply(lambda x: x.lstrip().startswith('RN_')))[0]
length = rns[1] - rns[0]
index = df[0].iloc[4:length]
cols = df[0][::length].apply(lambda x: x.lstrip()).values
result_df = pd.DataFrame(index=index)
for col_num, col_start in enumerate(range(0, len(df), length)):
    result_df[cols[col_num]] = df[1][col_num*length+4 : (col_num+1)*length].values
print result_df

输出：

     RN_48865   RN_7445   RN_9288  RN_10955
1    0.000014  0.000013  0.000011  0.000014
91   0.000014  0.000013  0.000011  0.000014
182  0.000014  0.000013  0.000011  0.000014
274  0.000014  0.000013  0.000011  0.000014
366  0.000014  0.000013  0.000011  0.000014
457  0.000014  0.000013  0.000011  0.000014
548  0.000014  0.000013  0.000011  0.000014

日期使用：

pandas.read_csv('file',
  parse_date=0,  # 0th column
  date_parser=lambda x: pandas.Timestamp('1995-1-1')+timedelta(x))

【讨论】：

感谢您对一个元素的帮助。用户 cyborg 指出我同意的问题不清楚。
太好了。非常感谢。看起来我在那里走错了路。还有很多东西要学。