如何在python中解析多行catalina日志-正则表达式

Question

我有 catalina 日志：

oct 21, 2016 12:32:13 AM org.wso2.carbon.identity.sso.agent.saml.SSOAgentHttpSessionListener sessionCreated
WARNING: HTTP Session created without LoggedInSessionBean
oct 21, 2016 3:03:20 AM com.sun.jersey.spi.container.ContainerResponse logException
SEVERE: Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException
    at ais.api.rest.rdss.Resource.lookAT(Resource.java:22)
    at sun.reflect.GeneratedMethodAccessor3019.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

我尝试在 python 中解析它。我的问题是我不知道日志中有多少行。最少为 2 行。我尝试从文件中读取，当第一行以 j、m、s、o 等开头时，这意味着它是日志的第一行，因为这是月份的第一个字母。但我不知道如何继续。当我停止阅读这些台词时？下一行何时以这些字母之一开头？但是我该怎么做呢？

import datetime
import re

SPACE = r'\s'
TIME = r'(?P<time>.*?M)'
PATH = r'(?P<path>.*?\S)'
METHOD = r'(?P<method>.*?\S)'
REQUEST = r'(?P<request>.*)'
TYPE = r'(?P<type>.*?\:)'

REGEX = TIME+SPACE+PATH+SPACE+METHOD+SPACE+TYPE+SPACE+REQUEST

def parser(log_line):
  match = re.search(REGEX,log_line)
    return ( (match.group('time'),
          match.group('path'), 
                              match.group('method'),
                              match.group('type'),
                              match.group('request')
                             )
                           )

db = MySQLdb.connect(host="localhost", user="myuser", passwd="mypsswd", db="Database")

with db:
  cursor = db.cursor()


    with open("Mylog.log","rw") as f:
        for line in f:

          if (line.startswith('j')) or (line.startswith('f')) or (line.startswith('m')) or (line.startswith('a')) or (line.startswith('s')) or (line.startswith('o')) or (line.startswith('n')) or (line.startswith('d')) :

          logLine = line
          result = parser(logLine)

                sql = ("INSERT INTO ..... ")
                data = (result[0])
                cursor.execute(sql, data)

f.close()
db.close()

我的最佳想法是一次只阅读两行。但这意味着丢弃所有其他数据。一定有更好的办法。

我想读这样的行： 1.line - oct 21, 2016 12:32:13 AM org.wso2.carbon.identity.sso.agent.saml.SSOAgentHttpSessionListener sessionCreated WARNING: HTTP Session created without LoggedInSessionBean

2.line - oct 21, 2016 3:03:20 AM com.sun.jersey.spi.container.ContainerResponse logException SEVERE: Mapped exception to response: 500 (Internal Server Error) javax.ws.rs.WebApplicationException at ais.api.rest.rdss.Resource.lookAT(Resource.java:22) at sun.reflect.GeneratedMethodAccessor3019.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl java:43)

3.line - oct 21, 2016 12:32:13 AM org.wso2.carbon.identity.sso.agent.saml.SSOAgentHttpSessionListener sessionCreated WARNING: HTTP Session created without LoggedInSessionBean

所以我想在行以 datetime 开头时开始阅读（这没问题）。问题是我想在下一行以日期时间开始时停止阅读。

Answer 1

这可能是你想要的。

我从生成器内的日志中读取行，以便确定它们是日期时间行还是其他行。另外，重要的是，我可以在日志文件中标记已到达文件结尾。

在程序的主循环中，当我获得日期时间行时，我开始在列表中累积行数。我第一次看到日期时间行时，如果它不为空，我会打印出来。由于程序将在文件结束时累积完整的行，因此我也安排在该点打印累积的行。

import re

a_date, other, EOF = 0,1,2

def One_line():
    with open('caroline.txt') as caroline:
        for line in caroline:
            line = line.strip()
            m = re.match(r'[a-z]{3}\s+[0-9]{1,2},\s+[0-9]{4}\s+[0-9]{1,2}:[0-9]{2}:[0-9]{2}\s+[AP]M', line, re.I)
            if m:
                yield a_date, line
            else:
                yield other, line
    yield EOF, ''

complete_line = []
for kind, content in One_line():
    if kind in [a_date, EOF]:
        if complete_line:
            print (' '.join(complete_line ))
        complete_line = [content]
    else:
        complete_line.append(content)

输出：

oct 21, 2016 12:32:13 AM org.wso2.carbon.identity.sso.agent.saml.SSOAgentHttpSessionListener sessionCreated WARNING: HTTP Session created without LoggedInSessionBean
oct 21, 2016 3:03:20 AM com.sun.jersey.spi.container.ContainerResponse logException SEVERE: Mapped exception to response: 500 (Internal Server Error) javax.ws.rs.WebApplicationException at ais.api.rest.rdss.Resource.lookAT(Resource.java:22) at sun.reflect.GeneratedMethodAccessor3019.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)