Python：文本日志文件处理并将行转换为列

Question

我是 python 新手，并且坚持使用文本格式的日志文件，它具有以下重复结构，我需要从行中提取数据并根据数据将其更改为列。例如

前 50 行是垃圾，如下所示（前六行）：

    ------------------------------------------------------------- 
Logging to file  xyz.
Char 
1,
 3 
r
 =

 ---------------------------------------------- 
Pid                             0 
Name                                   SAB=1, XYZ=3
---------------------------------------------- 
a              1 
b              2 
c              3
---------------------------------------------- 
Pid                             0 
Name                                   SAB=1, XYZ=3, P_NO=546467
---------------------------------------------- 
Test_data_1              00001 
Test_data_2              FOXABC 
Test_data_3         SHEEP123
Country             US
---------------------------------------------- 
Pid                             0 
Name                                   SAB=1
---------------------------------------------- 
Sno                  893489423

Log FileFormat

------------再继续一百万行。

现在需要的输出如下：

Required output format

PID, Name,       a,b,c
0, "SAB=1, XYZ=3", 1,2,3

PID, Name         , Test_data_1, Test_data_2, Test_data_3, Country
0, "SAB=1, XYZ=3, P_NO=546467", 00001, FOXABC, SHEEP123, US

Pid, Name, Sno
0, SAB=1, 893489423

我尝试编写代码但未能获得预期的结果：我的尝试如下：

'''
fn=open(file_name,'r')
for i,line in enumerate(fn ):
   if i >= 50 and "Name " in line:   # for first 50 line deletion/or starting point
         last_tag=line.split(",")[-1]
         last_element=last_tag.split("=")[0]
         print(last_element)    

'''

任何帮助将不胜感激。

Newly Discovered Structure

RBY Structure

Answer 1

我想出的解决方案有点乱，但确实有效，请查看以下内容：

import sys
import re
import StringIO


ifile = open(sys.argv[1],'r')   #Input log file as command-line argument
ofile = open(sys.argv[1][:-4]+"_formatted.csv",'w') #output formatted log txt

stringOut = ""

i = 0
flagReturn = True
j = 0

reVal = re.compile("Pid[\s]+(.*)\nName[\s]+(.*)\n[-]+\<br\>(.*)\<br\>") #Regex pattern for separating the Pid & Name from the variables
reVar = re.compile("(.*)[ ]+(.*)") #Regex pattern for getting vars and their values
reVarStr = re.compile(">>> [0-9]+.(.*)=(.*)") #Regex Pattern for Struct
reVarStrMatch = re.compile("Struct(.*)+has(.*)+members:") #Regex pattern for Struct check


for lines in ifile.readlines():
    if(i>8): #Omitting the first 9 lines of Garbage values
        if(lines.strip()=="----------------------------------------------"): #Checking for separation between PID & Name group and the Var group
            j+=1 #variable keeping track of whether we are inside the vars section or not (between two rows of hyphens)
            flagReturn = not flagReturn #To print the variables in single line to easily separate them with regex pattern reVal

        if(not flagReturn):
            stringTmp = lines.strip()+"<br>" #adding break to the end of each vars line in order for easier separation
        else:
            stringTmp = lines #if not vars then save each line as is

        stringOut += stringTmp #concatenating each lines to form the searchable string

    i+=1 #incrementing for omitting lines (useless after i=8)

    if(j==2):   #Once a complete set of PIDs, Names and Vars have been collected
        j=0     #Reset j
        matchObj = reVal.match(stringOut) #Match for PID, Name & Vars
        line1 = "Pid,Name,"
        line2 = matchObj.group(1).strip()+",\""+matchObj.group(2)+"\","
        buf = StringIO.StringIO(matchObj.group(3).replace("<br>","\n"))
        structFlag = False
        for line in buf.readlines(): #Separate each vars and add to the respective strings for writing to file
            if(not (reVarStrMatch.match(line) is None)):
                structFlag = True
            elif(structFlag and (not (reVarStr.match(line) is None))):
                matchObjVars = reVarStr.match(line)
                line1 += matchObjVars.group(1).strip()+","
                line2 += matchObjVars.group(2).strip()+","

            else:
                structFlag = False
                matchObjVars = reVar.match(line)
                try:
                    line1 += matchObjVars.group(1).strip()+","
                    line2 += matchObjVars.group(2).strip()+","
                except:
                    line1 += line.strip()+","
                    line2 += " ,"

        ofile.writelines(line1[:-1]+"\n")
        ofile.writelines(line2[:-1]+"\n")
        ofile.writelines("\n")
        stringOut = "" #Reseting the string 

ofile.close()
ifile.close()   

编辑 这也是我想出的包含新模式的方法。

我建议您执行以下操作：

1. 在日志文件的副本上运行解析器脚本，看看接下来哪里会失败。
2. 识别并记下破坏解析器的新模式。
3. 删除新识别的模式中的所有数据。
4. 从第 1 步开始重复，直到识别出所有模式。
5. 为每种类型的模式创建单独的正则表达式模式，并在单独的函数中调用它们以写入字符串。

编辑 2

structFlag = False
RBYflag = False
for line in buf.readlines(): #Separate each vars and add to the respective strings for writing to file
            if(not (reVarStrMatch.match(line) is None)):
                structFlag = True
            elif(structFlag and (not (reVarStr.match(line) is None))):
                matchObjVars = reVarStr.match(line)
                if(matchObjVars.group(1).strip()=="RBY" and not RBYFlag):
                    line1 += matchObjVars.group(1).strip()+","
                    line2 += matchObjVars.group(2).strip()+"**"
                    RBYFlag = True
                elif(matchObjVars.group(1).strip()=="RBY"):
                    line2 += matchObjVars.group(2).strip()+"**"
                else:
                    if(RBYFlag):
                        line2 = line2[:-2]
                        RBYFlag = False
                    line1 += matchObjVars.group(1).strip()+","
                    line2 += matchObjVars.group(2).strip()+","

        else:
            structFlag = False
            if(RBYFlag):
                line2 = line2[:-2]
                RBYFlag = False
            matchObjVars = reVar.match(line)
            try:
                line1 += matchObjVars.group(1).strip()+","
                line2 += matchObjVars.group(2).strip()+","
            except:
                line1 += line.strip()+","
                line2 += " ,"`

注意 这个循环已经变得非常臃肿，最好创建一个单独的函数来识别数据的类型并相应地返回一些值。