Python，在目录中所有 csv 文件的第一列中获取重复项

Question

import pandas as pd
import glob

dataset = pd.read_csv('masterfeedproduction-EURNA_2016-06-27.csv',sep = 
',',delimiter = None) # select 1 file in the directory
datasets_cols = ['transactionID','gvkey','companyName']

df= dataset.transactionID
df.shape
df.loc[df.duplicated()] 

返回所选文件中的副本。显示行号和事务ID。所以这是正确的。

target_directory = r'C:\Users\nikol\Downloads\fullDailyDeltas\fullDailyDeltas'
file_list = glob.glob(target_directory + "/*.csv")

df_result = df.loc[df.duplicated()]

for file in file_list:
   return(df_result) 

我卡住了。

target_directory = r'C:\Users\nikol\Downloads\fullDailyDeltas\fullDailyDeltas'
file_list = glob.glob(target_directory + "/*.csv")


for file in file_list:
    dataset = pd.read_csv(file)
    df = dataset.transactionID
    duplicated = df.loc[df.duplicated()]
    if duplicated.empty == False:
        print(file)
        print(duplicated)

Answer 1

看看glob 模块。

import pandas as pd
import glob

def your_function(file):   
    # put your df processing logic here
    return df_result

第 1 步 - 在目录中创建文件列表

target_directory = r'Path/to/your/dir'
file_list = glob.glob(target_directory + "/*.csv") 
# Include slash or it will search in the wrong directory!!

第 2 步 - 遍历列表中的文件

for file in file_list:                # Loop files
    df_result = your_function(file)   # Put your logic into a separate function
    new_filename = file.replace('.csv', '_processed.csv')
    df_result.to_csv(new_filename, index = False)

评论

如果您包含显示您自己尝试执行此操作的代码，您的问题会在几秒钟内得到回答。