使用正则表达式进行日志解析

Question

我正在尝试使用 Python 使用正则表达式解析 Apache 日志并将其分配给单独的变量。

ACCESS_LOG_PATTERN = '^(\S+) (\S+) (\S+) \[([\w:/]+\s[+\-]\d{4})\] "(\S+) (\S+)\s*(\S+)\s*" (\d{3}) (\S+)'

logLine='127.0.0.1 - - [01/Jul/1995:00:00:01 -0400] "GET /images/launch-logo.gif HTTP/1.0" 200 1839'

我将其解析并分组到以下变量中：

match = re.search(APACHE_ACCESS_LOG_PATTERN, logLine)



    host          = match.group(1)

    client_identd = match.group(2)

    user_id       = match.group(3)

    date_time     = match.group(4)

    method        = match.group(5)

    endpoint      = match.group(6)

    protocol      = match.group(7)

    response_code = int(match.group(8))

    content_size  = match.group(9)

正则表达式模式在日志行中运行良好，但在以下情况下解析/正则表达式匹配失败：

'127.0.0.1 - - [01/Jul/1995:00:00:01 -0400] "GET /" 200 1839'

'127.0.0.1 - - [01/Jul/1995:00:00:01 -0400] "GET / " 200 1839'

我该如何解决这个问题？

Answer 1

您需要通过添加? 使您的group 7 成为可选。使用以下正则表达式：

^(\S+) (\S+) (\S+) \[([\w:/]+\s[+\-]\d{4})\] "(\S+) (\S+)\s*(\S+)?\s*" (\d{3}) (\S+)
                                                                 ↑

见DEMO

Answer 2

import re


HOST = r'^(?P<host>.*?)'
SPACE = r'\s'
IDENTITY = r'\S+'
USER = r'\S+'
TIME = r'(?P<time>\[.*?\])'
REQUEST = r'\"(?P<request>.*?)\"'
STATUS = r'(?P<status>\d{3})'
SIZE = r'(?P<size>\S+)'

REGEX = HOST+SPACE+IDENTITY+SPACE+USER+SPACE+TIME+SPACE+REQUEST+SPACE+STATUS+SPACE+SIZE+SPACE

def parser(log_line):
    match = re.search(REGEX,log_line)
    return ( (match.group('host'),
            match.group('time'), 
                      match.group('request') , 
                      match.group('status') ,
                      match.group('size')
                     )
                   )


logLine = """180.76.15.30 - - [24/Mar/2017:19:37:57 +0000] "GET /shop/page/32/?count=15&orderby=title&add_to_wishlist=4846 HTTP/1.1" 404 10202 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"""
result = parser(logLine)
print(result)

结果

('180.76.15.30', '[24/Mar/2017:19:37:57 +0000]', 'GET /shop/page/32/?count=15&orderby=title&add_to_wishlist=4846 HTTP/1.1', '404', '10202')