使用 pandas 将包含许多表的 CSV 文件转换为一个数据框

Question

我想解释捆绑在 CSV 文件中的机器的数据输出。我想输出一张更容易分析的大表。

输入数据可能如下所示：

marker,
info 1, info 2 \r\n
col1,col2,col3,col4,col5,colA,colB,colC \r\n
NULL,2e15,-222,info,string,0.17,b,c \r\n
... \r\n
marker,
info 3, info 4 \r\n
col1,col5,colA,colB,colD \r\n
text,foo,0.17,-1e-12,string \r\n

换句话说，有逗号分隔的数据表，可以通过一些描述以下内容的标记字符串来区分。这些表的列比那些信息行多。表格具有列标题，并且每个表格的标题都不相同。这些表的行数不相等。

我希望输出将表格信息合并到单独的列中，并将所有不同的表格列连接起来。

我当前的代码工作正常，但感觉很粗糙，我想知道我是否可以直接从 pandas.read_csv() 完成这一切：

f = "test.csv"
locfile = locfile = open(f)
d={} #create an empty dictionary to load dataframes into
j=0 #I'm using this counter to find out how many etch tables there are in each CSV file
read_df = pd.read_csv(f, header=None, sep='\n') #read out the data in pandas 
test_df = read_df.loc[read_df[0].str.contains('marker')] #use .loc to find the expression separating the tables
step_holder = list(test_df.index.values) #store the first line of each table
step_holder.append(read_df.shape[0]) #add the bottom line of the last table
difference_step = [j-i for i, j in zip(step_holder[:-1], step_holder[1:])]#determine table length for each table
for line in locfile:
    if len(line)>1 and line.split(",")[0] == "marker": #here the header information is collected
        nextline= next(locfile)
        nextline_row = re.split(';|,|\*|\n',nextline)
        info1 = nextline_row[0]
        info2 = nextline_row[1] 
        header_info = [info1,info2]
        j+=1      
    if len(line)>1 and line.split(",")[0] == "col1": #here the actual process information is collected
        line_split = re.split(';|,|\*',line) #this first line will form the header information
        headers = ["info1","info2"]
        headers.extend(line_split)
        l = [] #create an empty list for storing all the rows in the data table
        for k in range(difference_step[j-1]): 
            row = []
            nextline= next(locfile) 
            nextline_row = re.split(';|,|\*',nextline)
            row = header_info+nextline_row
            l.append(row)            
        df = pd.DataFrame(l,columns=headers) #turn the list of rows into a dataframe
        d[f+str(j)] = df #add each dataframe to the dict to store them with unique IDs     
df = pd.concat([v for k,v in d.items()]) #concatenate all the dataframes in the dictionary
df.to_csv(outputf) 

有没有办法通过直接在 pandas 中重新排列数据来避免这种 for 循环？

Answer 1

好的，@Adrian 向我展示了方法，我只需 5 行代码就可以替换整个例程。

file_loc = "test.csv"
list = ['marker']
read_df = pd.read_csv(file_loc, header=None, names=range(60)) #read in the csv file. names tells python how many columns to expect
groups = read_df[0].isin(list).cumsum() #store the recurrence of 'marker'
tables = {'process'+str(k): g.iloc[0:] for k,g in read_df.groupby(groups)} #make a dictionary of dataframes

然后，我使用 for 循环来构建 tables 中的数据帧。