合并 csv 文件时添加文件名作为标题

Question

想要将所有 csv 文件合并到一个文件夹中。这按预期工作。

import os
import glob
import pandas as pd

extension = 'csv'
all_filenames = [i for i in glob.glob('*.{}'.format(extension))]

#combine all files in the list
combined_csv = pd.concat([pd.read_csv(f) for f in all_filenames], axis = 1)
#export to csv
combined_csv.to_csv( "combined.matrix", index=False)

但是我想将不带扩展名的文件名添加为标题。

文件1.csv

A,B
1,2
3,4

文件2.csv

A,B
5,6

combined.matrix

File1,File1,File2,File2
A,B,A,B
1,2,5,6
3,4,,

Answer 1

试试下面的代码：

import pandas as pd

all_filenames = ['File1.csv','File2.csv']
headers = []
for i in all_filenames:
    headers.append(i.replace('.csv', ''))

combined_csv = pd.concat([pd.read_csv(f) for f in all_filenames], keys=headers, axis = 1)

创建了一个标题列表，其中包含不包括扩展名的文件名。将列表传递给 pd.concat 函数中的 keys 参数。

Answer 2

基本思想是您可以在 DataFrame 本身的某处包含文件名（在这种情况下，我将其包含在列名中，您也可以将它们包含在一行中），因为无论如何您都在导出它到 csv 以供进一步处理

all_filenames = [i for i in glob.glob('*.{}'.format(extension))]
# This takes the value ["file1.csv", "file2.csv"]

#combine all files in the list
combined_csv = pd.concat([pd.read_csv(f) for f in all_filenames], axis = 1)
# This looks like
# A B A B
# 1 2 5 6
# 3 4 nan nan

由于列名是固定的（A 和 B） - 而且您对文件名更感兴趣，您可以更改列

combined_csv.columns = sorted(all_filenames * len(combined_csv.columns) / len(all_filenames))
# This evaluates to sorted(["file1.csv", "file2.csv"] * 4 / 2) which is equal to ["file1.csv", "file1.csv", "file2.csv", "file2.csv"]

现在您的数据框看起来像 - 表明哪一列来自哪个文件

# file1.csv file1.csv file2.csv file2.csv
# 1 2 5 6
# 3 4 nan nan

你可以导出到combined.matrix.csv

Answer 3

import os
import pandas as pd

parent_dir = 'YOUR_PARENT_DIRECTORY_PATH'
ext = 'csv'

combined_csv = pd.DataFrame()

for root, dir, files in os.walk(parent_dir):
    for f in files:
        path = os.path.join(root, f)
        filename, extension = os.path.splitext(f)
        if extension == f'.{ext}':
            new_df = pd.read_csv(path)
            cols = new_df.columns
            new_cols = []
            for c in cols:
                new_cols.append(f'{filename}{c}')
            new_df.columns = new_cols
            combined_csv = pd.concat([combined_csv, new_df], axis=1)

combined_csv.to_csv( "combined.matrix", index=False)