Python pandas - 将具有多个日期索引的csv合并到单个日期索引

Question

您好，我在电子表格中有如下数据

|aaa-date  |aaa-val|bbb-date  |bbb-val|ccc-date  |ccc-val|
|----------|-------|----------|-------|----------|-------|
|08-04-2008|-20.943|31-03-2008|-23.869|26-03-2008|+1.401 |
|09-04-2008|-20.943|01-04-2008|-19.813|27-03-2008|+1.376 |
|10-04-2008|-18.868|02-04-2008|-18.929|28-03-2008|-0.534 |
|11-04-2008|-19.057|03-04-2008|-19.917|31-03-2008|+0.688 |
|14-04-2008|-20.000|04-04-2008|-20.125|01-04-2008|+3.336 |
|15-04-2008|-18.868|07-04-2008|-21.321|02-04-2008|+3.413 |
|16-04-2008|-16.226|08-04-2008|-22.517|03-04-2008|+4.177 |
|17-04-2008|-14.340|09-04-2008|-24.857|04-04-2008|+4.279 |
|18-04-2008|-12.830|10-04-2008|-24.701|07-04-2008|+2.445 |
|21-04-2008|-15.472|11-04-2008|-24.857|08-04-2008|+1.146 |

我想导入这个（csv 或 xlsx）并到达一个只有一个日期索引和 aaa-val、bbb-val 列的数据框和 ccc-val 例如

|          |aaa-val|bbb-val|ccc-val|
|----------|-------|-------|-------|
|26-03-2008|       |       |+1.401 |
|27-03-2008|       |       |+1.376 |
|28-03-2008|       |       |-0.534 |
|31-03-2008|       |-23.869|+0.688 |
|01-04-2008|       |-19.813|+3.336 |
|02-04-2008|       |-18.929|+3.413 |
|03-04-2008|       |-19.917|+4.177 |
|04-04-2008|       |-20.125|+4.279 |
|07-04-2008|       |-21.321|+2.445 |
|08-04-2008|-20.943|-22.517|+1.146 |
|09-04-2008|-20.943|-24.857|+0.917 |
|10-04-2008|-18.868|-24.701|+2.420 |
|11-04-2008|-19.057|-24.857|+1.860 |
|14-04-2008|-20.000|-26.053|+3.515 |
|15-04-2008|-18.868|-24.805|       |
|16-04-2008|-16.226|-23.557|       |
|17-04-2008|-14.340|-23.765|       |
|18-04-2008|-12.830|       |       |
|21-04-2008|-15.472|       |       |
|22-04-2008|-16.793|       |       |
|23-04-2008|-13.019|       |       |
|24-04-2008|-12.453|       |       |
|25-04-2008|-12.642|       |       | 

除了加载到临时框架然后循环遍历日期/值列对之外，还有其他聪明的方法吗？

谢谢

Answer 1

所以 FWIW 这就是我最终的结果 - 我的数据集是 176 列 x 3300 行，concat 和 axis=1 似乎比 merge 更快

df = pd.read_csv('data.csv')
i = 0
new_df = pd.DataFrame()

while 2*(i+1) < len(df.columns):
    colname = df.columns[2*i + 1]
    tmp = df.iloc[:,[2*i, 2*i+1]].dropna()
    tmp.columns.values[0]='date'
    tmp.set_index('date', inplace=True)
    new_df = pd.concat([new_df, tmp], axis=1)
    i += 1

观察：

1. 我认为您无法避免循环遍历初始数据框 - 我找不到可以帮助您的 pandas 函数

2. iloc[:,[2*i, 2*i+1]] 是提取感兴趣的列的超级有用的构造 - 这可能对新手 How to take column slices of a Pandas DataFrame in Python 有帮助

谢谢大家，约翰

Answer 2

您可以先分离数据框，然后将它们合并...：

data_csv = io.StringIO('''|aaa-date  |aaa-val|bbb-date  |bbb-val|ccc-date  |ccc-val|
|08-04-2008|-20.943|31-03-2008|-23.869|26-03-2008|+1.401 |
|09-04-2008|-20.943|01-04-2008|-19.813|27-03-2008|+1.376 |
|10-04-2008|-18.868|02-04-2008|-18.929|28-03-2008|-0.534 |
|11-04-2008|-19.057|03-04-2008|-19.917|31-03-2008|+0.688 |
|14-04-2008|-20.000|04-04-2008|-20.125|01-04-2008|+3.336 |
|15-04-2008|-18.868|07-04-2008|-21.321|02-04-2008|+3.413 |
|16-04-2008|-16.226|08-04-2008|-22.517|03-04-2008|+4.177 |
|17-04-2008|-14.340|09-04-2008|-24.857|04-04-2008|+4.279 |
|18-04-2008|-12.830|10-04-2008|-24.701|07-04-2008|+2.445 |
|21-04-2008|-15.472|11-04-2008|-24.857|08-04-2008|+1.146 |''')
df = pd.read_csv(data_csv,sep=r'\s*\|\s*',engine='python').iloc[:,1:-1]
column_names = df.columns.tolist()
cols = [col.split('-')[0] for col in column_names][::2]
cols
dfs = [df[[col+'-date',col+'-val']] for col in cols]
new_df = pd.DataFrame({'date':[]})
for dfi,col in zip(dfs,column_names[::2]):
    new_df = new_df.merge(dfi.rename(columns={col:'date'}),how='outer')
new_df

输出：

    date        aaa-val bbb-val ccc-val
0   08-04-2008  -20.943 -22.517 1.146
1   09-04-2008  -20.943 -24.857 NaN
2   10-04-2008  -18.868 -24.701 NaN
3   11-04-2008  -19.057 -24.857 NaN
4   14-04-2008  -20.000 NaN     NaN
5   15-04-2008  -18.868 NaN     NaN
6   16-04-2008  -16.226 NaN     NaN
7   17-04-2008  -14.340 NaN     NaN
8   18-04-2008  -12.830 NaN     NaN
9   21-04-2008  -15.472 NaN     NaN
10  31-03-2008  NaN     -23.869 0.688
11  01-04-2008  NaN     -19.813 3.336
12  02-04-2008  NaN     -18.929 3.413
13  03-04-2008  NaN     -19.917 4.177
14  04-04-2008  NaN     -20.125 4.279
15  07-04-2008  NaN     -21.321 2.445
16  26-03-2008  NaN     NaN     1.401
17  27-03-2008  NaN NaN 1.376
18  28-03-2008  NaN NaN -0.534

Answer 3

我只是在查找其他内容时发现了这篇文章，我相信它可以帮助您：

https://pbpython.com/pandas-excel-range.html

基本上，您可以为每个时间/数据范围读取特定列范围的文件（如果您想使用列名，请使用 lambda 方法）。然后我会将日期字段重命名为相同的名称或将日期字段设置为索引。然后到多个全外连接来组合所有数据。

编辑 - 一个简单的 concat 不会像我最初写的那样工作。我建议在日期列上使用完整的外部联接。

[来自链接]

使用可调用对象的另一种方法是包含 lambda 表达式。这是一个示例，我们只想包含已定义的列列表。为了比较，我们通过将名称转换为小写来规范化名称。

cols_to_use = ['item_type', 'order id', 'order date', 'state', 'priority']
df = pd.read_excel(src_file,
                   header=1,
                   usecols=lambda x: x.lower() in cols_to_use)

编辑显示 CONCAT 和 MERGE 之间的差异：

import pandas as pd
import numpy as np
from common import  show_table

df1 = pd.DataFrame(data=[[1, 1], [2, 2]], columns=['a','b'])
print(df1)
#    a  b
# 0  1  1
# 1  2  2

df2 = pd.DataFrame(data=[[1, 1], [3, 3]], columns=['a','c'])
print(df2)
#    a  c
# 0  1  1
# 1  3  3

# no good...
df3 = pd.concat([df1, df2])
print(df3)
#    a    b    c
# 0  1  1.0  NaN
# 1  2  2.0  NaN
# 0  1  NaN  1.0
# 1  3  NaN  3.0


# good
df4 = pd.merge(df1, df2, how='outer', on='a')
print(df4)
#    a    b    c
# 0  1  1.0  1.0
# 1  2  2.0  NaN
# 2  3  NaN  3.0

编辑索引验证 - 索引上的 Concat 不执行完全外连接

import pandas as pd
import numpy as np

df1 = pd.DataFrame(data=[[1, 1], [2, 2]], columns=['a','b'])
df1 = df1.set_index('a')
print(df1)
#    b
# a   
# 1  1
# 2  2
df2 = pd.DataFrame(data=[[1, 1], [3, 3]], columns=['a','c'])
df2 = df2.set_index('a')
print(df2)
#    c
# a   
# 1  1
# 3  3

# no good...
df3 = pd.concat([df1, df2])
print(df3)
#      b    c
# a          
# 1  1.0  NaN
# 2  2.0  NaN
# 1  NaN  1.0
# 3  NaN  3.0

# good
df4 = pd.merge(df1, df2, how='outer', left_index=True, right_index=True)
print(df4)
#      b    c
# a          
# 1  1.0  1.0
# 2  2.0  NaN
# 3  NaN  3.0