用python和pandas将CSV文件中的三列合二为一

Question

您好，我正在尝试将多个现有列合并为 1 个新列，然后删除 CSV 文件中的三个原始列。我一直在尝试用熊猫来做到这一点，但运气不佳。我对 python 还很陌生。

我的代码首先将多个 CSV 文件组合在同一个目录中，然后尝试操作这些列。第一个组合有效，我得到一个包含组合数据的 output.csv，但是列的组合没有。

import glob
import pandas as pd

interesting_files = glob.glob("*.csv")

header_saved = False
with open('output.csv','wb') as fout:
    for filename in interesting_files:
        with open(filename) as fin:
            header = next(fin)
            if not header_saved:
                fout.write(header)
                header_saved = True
            for line in fin:
                fout.write(line)

df = pd.read_csv("output.csv")
df['HostAffected']=df['Host'] + "/" + df['Protocol'] + "/" + df['Port']
df.to_csv("newoutput.csv")

有效地改变这个：

Host,Protocol,Port
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,49707
10.0.0.10,tcp,49672
10.0.0.10,tcp,49670

变成这样：

HostsAffected
10.0.0.10/tcp/445
10.0.0.10/tcp/445
10.0.0.10/tcp/445
10.0.0.10/tcp/445
10.0.0.10/tcp/445
10.0.0.10/tcp/445
10.0.0.11/tcp/445
10.0.0.11/tcp/49707
10.0.0.11/tcp/49672
10.0.0.11/tcp/49670
10.0.0.11/tcp/49668
10.0.0.11/tcp/49667

但是 csv 中还有其他列。

我不是程序员，我只是想解决一个问题，非常感谢任何帮助。

Answer 1

在我看来，我们有三种选择：

%timeit df['Host'] + "/" + df['Protocol'] + "/" + df['Port'].map(str)
%timeit ['/'.join(i) for i in zip(df['Host'],df['Protocol'],df['Port'].map(str))]
%timeit ['/'.join(i) for i in df[['Host','Protocol','Port']].astype(str).values]

时间安排：

10 loops, best of 3: 39.7 ms per loop  
10 loops, best of 3: 35.9 ms per loop  
10 loops, best of 3: 162 ms per loop

无论多么慢，我认为这将是您最易读的方法：

import pandas as pd

data = '''\
ID,Host,Protocol,Port
1,10.0.0.10,tcp,445
1,10.0.0.10,tcp,445
1,10.0.0.10,tcp,445
1,10.0.0.10,tcp,445
1,10.0.0.10,tcp,445
1,10.0.0.10,tcp,445
1,10.0.0.10,tcp,445
1,10.0.0.10,tcp,49707
1,10.0.0.10,tcp,49672
1,10.0.0.10,tcp,49670'''

df = pd.read_csv(pd.compat.StringIO(data)) # Recreates a sample dataframe

cols = ['Host','Protocol','Port']
newcol = ['/'.join(i) for i in df[cols].astype(str).values]
df = df.assign(HostAffected=newcol).drop(cols, 1)
print(df)

返回：

   ID         HostAffected
0   1    10.0.0.10/tcp/445
1   1    10.0.0.10/tcp/445
2   1    10.0.0.10/tcp/445
3   1    10.0.0.10/tcp/445
4   1    10.0.0.10/tcp/445
5   1    10.0.0.10/tcp/445
6   1    10.0.0.10/tcp/445
7   1  10.0.0.10/tcp/49707
8   1  10.0.0.10/tcp/49672
9   1  10.0.0.10/tcp/49670

Answer 2

有几种方法可以做到这一点：使用矢量化函数来组合系列，或者使用 lambda 函数和 pd.Series.apply。

矢量化解

不要忘记将非数字类型转换为str。

df['HostAffected'] = df['Host'] + '/' + df['Protocol'] + '/' + df['Port'].map(str)

性能说明：Converting a series of ints to strings - Why is apply much faster than astype?

应用lambda函数

df['HostsAffected'] = df.apply(lambda x: '/'.join(list(map(str, x))), axis=1)

使用这两种解决方案，您只需按此列过滤即可删除所有其他解决方案：

df = df[['HostsAffected']]

完整示例

from io import StringIO
import pandas as pd

mystr = StringIO("""Host,Protocol,Port
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,49707
10.0.0.10,tcp,49672
10.0.0.10,tcp,49670""")

# replace mystr with 'file.csv'
df = pd.read_csv(mystr)

# combine columns
df['HostsAffected'] = df['Host'] + '/' + df['Protocol'] + '/' + df['Port'].map(str)

# include only new columns
df = df[['HostsAffected']]

结果：

print(df)

         HostsAffected
0    10.0.0.10/tcp/445
1    10.0.0.10/tcp/445
2    10.0.0.10/tcp/445
3    10.0.0.10/tcp/445
4    10.0.0.10/tcp/445
5    10.0.0.10/tcp/445
6    10.0.0.10/tcp/445
7  10.0.0.10/tcp/49707
8  10.0.0.10/tcp/49672
9  10.0.0.10/tcp/49670

Answer 3

你可以这样做：

    dt = """Host,Protocol,Port
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,445
10.0.0.10,tcp,49707
10.0.0.10,tcp,49672
10.0.0.10,tcp,49670"""

tdf = pd.read_csv(pd.compat.StringIO(dt))
tdf['HostsAffected'] = tdf.apply(lambda x: '{}/{}/{}'.format(x['Host'] , x['Protocol'] , x['Port']), axis=1)
tdf = tdf[['HostsAffected']]
tdf.to_csv(<path-to-save-csv-file>)

这将是输出：

    HostsAffected
0   10.0.0.10/tcp/445
1   10.0.0.10/tcp/445
2   10.0.0.10/tcp/445
3   10.0.0.10/tcp/445
4   10.0.0.10/tcp/445
5   10.0.0.10/tcp/445
6   10.0.0.10/tcp/445
7   10.0.0.10/tcp/49707
8   10.0.0.10/tcp/49672
9   10.0.0.10/tcp/49670

如果您正在从文件中读取 CSV，请按如下方式编辑 read_csv 行：

tdf = pd.read_csv(<path-to-the-file>)