Sophos syslog 的 grok 表达式

Question

我在尝试找出以下消息类型的 grok 表达式时遇到了很多麻烦（来自 Sophos UTM）

2018 年 4 月 28 日 16:57:49 utm-vap-xx.domain.local: 04:28-17:02:05 s-utm-01 httpproxy[52816]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="POST" srcip="10.11.110.5" dstip="216.163.176.36" user="" group="" ad_domain= "" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (默认 Web 过滤器配置文件)" filteraction="REF_DefaultHTTPCFFAction (默认内容过滤器操作)" size="15" request="0xdae2cc00" url="http:/ /iprep3.t.ctmail.com/SpamResolverNG/SpamResolverNG.dll?DoNewRequest"referer=""error=""authtime="0" dnstime="905" cattime="143" avscantime="2275" fullreqtime="238344" device="0" auth="0" ua="Mozilla/4.0 (compatible; Win32; Commtouch Http Client (curl))" exceptions="" category="178" credit="neutral" categoryname="Internet Services" country ="美国" content-type="text/html" 沙箱="-"

当我想跳过字段或值对包含空字符串时，就会出现问题。例如在 Logstash 添加类似

()?(srcip={%"IP:SourceIP"})

正在引起问题，而这些在在线 grok builder 中确实有效

目标是得到类似的东西

Sub
SourceIP
Destination IP
Protocol etc

我还打算在 Logstash 中使用我已经与其他来源合作的地理标签。

期待收到一些有价值的帮助。谢谢

Answer 1

有人写了Grok Sophos UTM 9.x Pattern for logstash，

Sophos UTM ulogd 和 pluto log 的极增强 Grok 模式 各种格式的消息，并将消息正文与 标题

 grok {
  pattern => ['(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGHOST:logsource}) (?:%{YEAR}): (?:%{MONTHNUM}):(?:%{MONTHDAY})-(?:%{HOUR}):(?:%{MINUTE}):(?:%{SECOND}) (?:%{SYSLOGHOST}) (?:%{SYSLOGPROG}): (?<messagebody>(?:id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" fwrule=\"%{INT:utm_ulogd_fwrule}\" initf=\"%{DATA:utm_ulogd_initf}\" outitf=\"%{DATA:utm_ulogd_outif}\" (?:srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\" dstmac=\"%{GREEDYDATA:utm_ulogd_dstmac}\"|srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\") srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" length=\"%{INT:utm_ulogd_pkglength}\" tos=\"%{DATA:utm_ulogd_tos}\" prec=\"%{DATA:utm_ulogd_prec}\" ttl=\"%{INT:utm_ulogd_ttl}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\" tcpflags=\"%{DATA:utm_ulogd_tcpflags}\"|id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" fwrule=\"%{INT:utm_ulogd_fwrule}\" initf=\"%{DATA:utm_ulogd_initf}\" outitf=\"%{DATA:utm_ulogd_outif}\" (?:srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\" dstmac=\"%{GREEDYDATA:utm_ulogd_dstmac}\"|srcmac=\"%{GREEDYDATA:utm_ulogd_srcmac}\") srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" length=\"%{INT:utm_ulogd_pkglength}\" tos=\"%{DATA:utm_ulogd_tos}\" prec=\"%{DATA:utm_ulogd_prec}\" ttl=\"%{INT:utm_ulogd_ttl}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\"|id=\"%{INT:utm_id}\" severity=\"%{LOGLEVEL:utm_severity}\" sys=\"%{DATA:utm_sys}\" sub=\"%{DATA:utm_sub}\" name=\"%{DATA:utm_name}\" action=\"%{DATA:utm_action}\" reason=\"%{DATA:utm_ips_reason}\" group=\"%{INT:utm_ips_group}\" srcip=\"%{IP:utm_srcip}\" dstip=\"%{IP:utm_dstip}\" proto=\"%{INT:utm_protocol}\" srcport=\"%{INT:utm_srcport}\" dstport=\"%{INT:utm_dstport}\" sid=\"%{INT:utm_ips_sid}\" class=\"%{DATA:utm_ips_class}\" priority=\"%{INT:utm_ips_priority}\" generator=\"%{INT:utm_ips_generator}\" msgid=\"%{INT:utm_ips_msgid}\"|\"%{DATA:utm_pluto_vpnname}\"\[%{INT}\] %{IP:utm_pluto_vpnremoteip} #%{INT}: %{GREEDYDATA:utm_pluto_message}|%{GREEDYDATA}))']
  type => "sophosutm"
 }

这适用于您的日志（我已经测试过）

但是，如果您对分隔为字段的整个数据不感兴趣，而只对问题中提到的特定数据感兴趣，那么您可以将不必要的数据分配为 GREEDYDATA 并仅提取所需的字段，如下所示，

sub=%{QUOTEDSTRING:protocol}\s*.*\s*srcip=\"%{IP:SourceIP}\"\s*dstip=\"%{IP:DestinationIP}\"

上面的grok模式会提取Sub、SourceIP和Destination IP，并产生如下输出，

 {
  "protocol": [
    [
      ""http""
    ]
  ],
  "SourceIP": [
    [
      "10.11.110.5"
    ]
  ],
  "IPV6": [
    [
      null,
      null
    ]
  ],
  "IPV4": [
    [
      "10.11.110.5",
      "216.163.176.36"
    ]
  ],
  "DestinationIP": [
    [
      "216.163.176.36"
    ]
  ]
}

可以使用相同的模式过滤更多数据。

Answer 2

这是我用来获取 modsecurity 消息的内容。我从来没有回去将表达式简化为一个，所以它们都被调用了，但这应该适用于你的 reverseproxy.log 或其他任何不是直接 KV 的东西。

if "ModSecurity:" in [message] {
    grok {
    break_on_match => false
        match => [
        "message", ' \[hostname %{QUOTEDSTRING:Hostname}\] \[client %{IPORHOST:Source_IP}\]'
        ] #end match
    } #end grok
     grok {
    break_on_match => false
        match => [
        "message", ' \[client %{IPORHOST:Source_IP}\]'
        ] #end match
    } #end grok
    grok {
    break_on_match => false
        match => [
        "message", ' \[severity %{QUOTEDSTRING:Rule_Severity}\]'
        ] #end match
    } #end grok
    grok {
    break_on_match => false
        match => [
        "message", ' \[id %{QUOTEDSTRING:Rule_ID}\]'
        ] #end match
    } #end grok
    grok {
    break_on_match => false
        match => [
        "message", ' \[uri %{QUOTEDSTRING:Target_URI}\]'
        ] #end match
         add_field => [ "Logsource" , "Reverse Proxy (Modsecurity)" ]
    } #end grok
    grok {
    break_on_match => false
        match => [
        "message", '\[msg %{QUOTEDSTRING:MSG}\]'
        ] #end match
        add_field => [ "received_at", "%{@timestamp}" ]
    } #end grok
    date {
        match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ]
    } #end date
    mutate {
    } # end mutate
  } #end "if ModSecurity:"