单个 CSV 输出，不同列中的数据

Question

我有许多 CSV 文件，其中的数据仅在前三列中。我想从每个 CSV 文件中复制数据并按列顺序将其粘贴到一个 CSV 文件中。例如，第一个 CSV 文件中的数据进入输出文件的第 1,2 和 3 列。同样，来自第 2 个 CSV 的数据进入同一输出 CSV 文件的第 4、5 和 6 列，依此类推。任何帮助将不胜感激。谢谢。

我已经尝试了以下代码，但它只能在相同的列中获得输出。

import glob
import pandas as pd
import time
import numpy as np
start = time.time()

Filename='Combined_Data.csv'

extension = 'csv'
all_filenames = [i for i in glob.glob('*.{}'.format(extension))]

for i in range(len(all_filenames)):
    data= pd.read_csv(all_filenames[i],skiprows=23)
    data= data.rename({'G1': 'CH1', 'G2': 'CH2','Dis': 'CH3'},axis=1) 
    data= data[['CH1','CH2','CH3']]
    data= data.apply(pd.to_numeric, errors='coerce')
    print(all_filenames[i])
    if i == 0:
    data.to_csv(Filename,sep=',',index=False,header=True,mode='a')
    else:
    data.to_csv(Filename,sep=',',index=False,header=False,mode='a')

end = time.time()
print((end - start),'Seconds(Execution Time)')

Answer 1

如果您不需要为此编写自己的代码，我推荐 GoCSV 的 zip 命令；它还可以处理具有不同行数的 CSV。

我有三个 CSV 文件：

file1.csv

Dig1,Dig2,Dig3
1,2,3
4,5,6
7,8,9

file2.csv

Letter1,Letter2,Letter3
a,b,c
d,e,f

和file3.csv

RomNum1,RomNum2,RomNum3
I,II,III

当我运行gocsv zip file2.csv file1.csv file3.csv 时，我得到：

Letter1,Letter2,Letter3,Dig1,Dig2,Dig3,RomNum1,RomNum2,RomNum3
a,b,c,1,2,3,I,II,III
d,e,f,4,5,6,,,
,,,7,8,9,,,

对于许多不同的操作系统，GoCSV 是 pre-built。

Answer 2

下面是如何使用 Python 的 CSV 模块，使用这些文件：

file1.csv

Dig1,Dig2,Dig3
1,2,3
4,5,6
7,8,9

file2.csv

Letter1,Letter2,Letter3
a,b,c
d,e,f

和file3.csv

RomNum1,RomNum2,RomNum3
I,II,III

更占用内存的选项

这会一次累积最终的 CSV 文件，并使用每个新的输入 CSV 扩展代表最终 CSV 的列表。

#!/usr/bin/env python3
import csv
import sys

csv_files = [
    'file2.csv',
    'file1.csv',
    'file3.csv',
]

all = []

for csv_file in csv_files:
    with open(csv_file) as f:
        reader = csv.reader(f)
        rows = list(reader)

        len_all = len(all)

        # First file, initialize all and continue (skip)
        if len_all == 0:
            all = rows
            continue

        # The number of columns in all so far
        len_cols = len(all[0])

        # Extend all with the new rows
        for i, row in enumerate(rows):
            # Check to make sure all has as many rows as this file
            if i >= len_all:
                all.append(['']*len_cols)

            all[i].extend(row)


# Finally, pad all rows on the right
len_cols = len(all[0])
for i in range(len(all)):
    len_row = len(all[i])
    if len_row < len_cols:
        col_diff = len_cols - len_row
        all[i].extend(['']*col_diff)


writer = csv.writer(sys.stdout)
writer.writerows(all)

流式传输选项

这一次读取和写入一行/行。

（这基本上是来自 GoCSV 的 zip 的 Go 代码的 Python 端口，来自上面）

import csv
import sys

fnames = [
    'file2.csv',
    'file1.csv',
    'file3.csv',
]
num_files = len(fnames)

readers = [csv.reader(open(x)) for x in fnames]

# Collect "header" lines; each header defines the number
# of columns for its file
headers = []
num_cols = 0
offsets = [0]
for reader in readers:
    header = next(reader)
    headers.append(header)
    num_cols += len(header)
    offsets.append(num_cols)

writer = csv.writer(sys.stdout)

# With all headers counted, every row must have this many columns
shell_row = [''] * num_cols

for i, header in enumerate(headers):
    start = offsets[i]
    end = offsets[i+1]
    shell_row[start:end] = header

# Write headers
writer.writerow(shell_row)

# Expect that not all CSVs have the same number of rows; some will "finish" ahead of others
file_is_complete = [False] * num_files
num_complete = 0

# Loop a row at a time...
while True:
    # ... for each CSV
    for i, reader in enumerate(readers):
        if file_is_complete[i]:
            continue

        start = offsets[i]
        end = offsets[i+1]
        try:
            row = next(reader)
            # Put this row in its place in the main row
            shell_row[start:end] = row
        except StopIteration:
            file_is_complete[i] = True
            num_complete += 1
        except:
            raise

    if num_complete == num_files:
        break

    # Done iterating CSVs (for this row), write it
    writer.writerow(shell_row)

    # Reset for next main row
    shell_row = [''] * num_cols

对于任何一个，我都得到：

Letter1,Letter2,Letter3,Dig1,Dig2,Dig3,RomNum1,RomNum2,RomNum3
a,b,c,1,2,3,I,II,III
d,e,f,4,5,6,,,
,,,7,8,9,,,