使用 Promtail 的多个条件将日志行删除到 Loki答案

【问题标题】：Drop log lines to Loki using multiple conditions with Promtail使用 Promtail 的多个条件将日志行删除到 Loki
【发布时间】：2022-12-28 14:05:04
【问题描述】：

我想使用来自两个不同 JSON 字段的 AND 条件在 Promtail 中删除行。

我有这样的 JSON 日志行。

{"timestamp":"2022-03-26T15:40:41+00:00","remote_addr":"1.2.3.4","remote_user":"","request":"GET / HTTP/1.1","status": "200","body_bytes_sent":"939","request_time":"0.000","http_referrer":"http://5.6.7.8","http_user_agent":"user agent 1"}  
{"timestamp":"2022-03-26T15:40:41+00:00","remote_addr":"1.2.3.4","remote_user":"","request":"GET /path HTTP/1.1","status": "200","body_bytes_sent":"939","request_time":"0.000","http_referrer":"http://5.6.7.8","http_user_agent":"user agent 1"}
{"timestamp":"2022-03-26T15:40:41+00:00","remote_addr":"1.2.3.4","remote_user":"","request":"GET / HTTP/1.1","status": "200","body_bytes_sent":"939","request_time":"0.000","http_referrer":"http://5.6.7.8","http_user_agent":"user agent 2"}

我的本地 Promtail 配置如下所示。

clients:
  - url: http://localhost:3100/loki/api/v1/push
scrape_configs:
  - job_name: testing-my-job-drop
    pipeline_stages:
      - match:
          selector: '{job="my-job"}'
          stages:
            - json:
                expressions:
                  http_user_agent:
                  request:
            - drop:
                source: "http_user_agent"
                expression: "user agent 1"
            # I want this to be AND
            - drop:
                source: "request"
                expression: "GET / HTTP/1.1"
          drop_counter_reason: my_job_healthchecks
    static_configs:
      - labels:
          job: my-job

使用像这样的 Promtail 配置从我的两个 JSON 字段中删除使用 OR 的行。

我怎样才能调整我的配置，以便我只删除 http_user_agent = user agent 1 和 request = GET / HTTP/1.1 的行？

【问题讨论】：

标签： grafana-loki promtail

【解决方案1】：

如果您提供多个选项，它们将被视为 AND 子句，其中每个选项都必须为真才能删除日志。如果您希望使用 OR 子句删除，请指定多个删除阶段。

https://grafana.com/docs/loki/latest/clients/promtail/stages/drop/#drop-stage

按时间或长度删除日志

将丢弃所有早于 24 小时或长于 8kb 字节的日志

- json:
    expressions:
     time:
     msg:
- timestamp:
    source: time
    format: RFC3339
- drop:
    older_than: 24h
- drop:
    longer_than: 8kb

按正则表达式和长度删除日志

将删除所有包含单词 debug 且长度超过 1kb 字节的日志

- drop:
    expression: ".*debug.*"
    longer_than: 1kb

【讨论】：

【解决方案2】：

clients:
  - url: http://localhost:3100/loki/api/v1/push
scrape_configs:
  - job_name: testing-my-job-drop
    pipeline_stages:
      - match:
          selector: '{job="my-job"}'
          stages:
            - json:
                expressions:
                  http_user_agent:
                  request:

            - match:
              selector: '{http_user_agent="user agent 1"}'
              stages:
              - drop:
                  source: "request"
                  expression: "GET / HTTP/1.1"
                  drop_counter_reason: my_job_healthchecks
              ## they are both conditions match will drop

    static_configs:
      - labels:
          job: my-job

match stage包括match stage。

【讨论】：