用于打印aws elb日志的基于python的logparser的正则表达式？答案

【问题标题】：regex for python based logparser for printing aws elb logs?用于打印aws elb日志的基于python的logparser的正则表达式？
【发布时间】：2021-10-22 19:06:36
【问题描述】：

我正在尝试编写 python 代码以从 elb 日志中提取某些字段，但我无法为所有 elb 日志字段（如 "user_agent"、request 等）找到正确的正则表达式

喜欢如何打印图案 "POST https://example.com:443/api/pages/uuids/8ad6e82e-f86b-11ea-a68d-cbc99f85d247/updateUserHeartbeat HTTP/2.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" 来自下方使用通用正则表达式的日志

这里提到了各种elb字段https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html

我得到的示例正则表达式：

regex = r'([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*):([0-9]*) ([^ ]*)[:-]([0-9]*) ([-.0-9]*) ([-.0-9]*) ([-.0-9]*) (|[-0-9]*) (-|[-0-9]*) ([-0-9]*) ([-0-9]*) \"([^ ]*) ([^ ]*) (- |[^ ]*)\" \"([^\"]*)\" ([A-Z0-9-]+) ([A-Za-z0-9.-]*) ([^ ]*) \"([^\"]*)\" \"([^\"]*)\" \"([^\"]*)\" ([-.0-9]*) ([^ ]*) \"([^\"]*)\" \"([^\"]*)\" \"([^ ]*)\" \"([^\s]+?)\" \"([^\s]+)\" \"([^ ]*)\" \"([^ ]*)\"'
line_split = re.split(regex, line)

日志文件中的示例日志行如下

h2 2021-06-07T23:57:13.300250Z app/megapool-retool-app/dbb257b8adaa87cf 93.107.2.244:59799 - -1 -1 -1 302 - 3087 561 "POST https://example.com:443/api/pages/uuids/8ad6e82e-f86b-11ea-a68d-cbc99f85d247/updateUserHeartbeat HTTP/2.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-west-2:752180062774:targetgroup/megapool-retool-app/1665e090211d92fc "Root=1-6089b259-1c8c6bca3b1d7a895a21a694" "xyz.com" "arn:aws:acm:us-west-2:75218123456562774:certificate/b7a45f0c-3009-42c2-97b9-ab81a61d1b25" 0 2021-06-07T23:57:13.299000Z "authenticate" "-" "-" "-" "-" "-" "-"

【问题讨论】：

标签： python regex unix logparser aws-elb

【解决方案1】：

您的示例正则表达式会获得很多您实际上并不想要的信息。将其限制在您想要的范围内，并使用您对文本的了解。

import re

finder = re.compile(r"\"(\w{3,4}) (\S*) ([^\"]*)\" \"([^\"]*)\"")

with open("testlog.txt", "r") as fp:
    txt = fp.read()

for req_type, url, protocol, browser_details in finder.findall(txt):
    print(f"{req_type=}")
    print(f"{url=}")
    print(f"{protocol=}")
    print(f"{browser_details=}")

【讨论】：

【解决方案2】：

我设法通过根据我使用的早期/原始正则表达式修改我的 python 代码中的字段列表来修复它，因为在基于正则表达式的代码中它也拆分了 client_ip 和 client_port 所以在修复字段列表之后，一切正常

下面是我的代码 sn-p 这段代码sn -p 对分析elb日志文件很有用，可以根据需要进一步修改

import re

fields = [ "type",
"time",
"elb",
"client_ip",
"client_port",
"target_ip",
"target_port",
"request_processing_time",
"target_processing_time",
"response_processing_time",
"elb_status_code",
"target_status_code",
"received_bytes",
"sent_bytes",
"request_type",
"request_url",
"request_protocol",
"user_agent_browser",
"ssl_cipher",
"ssl_protocol",
"target_group_arn",
"trace_id",
"domain_name",
"chosen_cert_arn",
"matched_rule_priority",
"request_creation_time",
"actions_executed",
"redirect_url",
"lambda_error_reason",
"target_port_list",
"target_status_code_list",
"classification",
"classification_reason" ]



field = str(input("what is the field needed? "))
regex = r'([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*):([0-9]*) ([^ ]*)[:-]([0-9]*) ([-.0-9]*) ([-.0-9]*) ([-.0-9]*) (|[-0-9]*) (-|[-0-9]*) ([-0-9]*) ([-0-9]*) \"([^ ]*) ([^ ]*) (- |[^ ]*)\" \"([^\"]*)\" ([A-Z0-9-]+) ([A-Za-z0-9.-]*) ([^ ]*) \"([^\"]*)\" \"([^\"]*)\" \"([^\"]*)\" ([-.0-9]*) ([^ ]*) \"([^\"]*)\" \"([^\"]*)\" \"([^ ]*)\" \"([^\s]+?)\" \"([^\s]+)\" \"([^ ]*)\" \"([^ ]*)\"'

def ParseLogFile(file):
    resultDict = {}

    with open(file, 'r') as log:
        line = log.readline()
        while line:
            line_split = re.split(regex, line)
            line_split = line_split[1:len(line_split) - 1]
            index = fields.index(field)
            val = line_split[index]
            resultDict.setdefault(val, 0)
            resultDict[val] += 1
            line = log.readline()
        return resultDict
if __name__ == '__main__':
    result=ParseLogFile("C:\\HOME\\2.log")
    print(result)

【讨论】：