Elastic Beanstalk：Cloudwatch 代理随机停止流式传输日志答案

【问题标题】：ElasticBeanstalk: Cloudwatch agent stops streaming logs randomelyElastic Beanstalk：Cloudwatch 代理随机停止流式传输日志
【发布时间】：2020-12-24 23:40:52
【问题描述】：

我通过 ElasticBeanstalk 部署了一个简单的 nodejs (express) 应用程序，我在配置中启用了 cloudwatch 日志，并且日志开始流式传输。问题是，它只是随机停止。我四处搜索并遇到this 答案，随后它重新启动了流媒体，但过了一段时间它又挂断了，我必须重新启动它。我如何首先防止它发生？

我什至尝试手动提供配置。这是我放在.ebextensions 目录下的.config 文件。之后没区别，这个样本取自here，直接文件link。

packages:
  yum:
    awslogs: []

files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`

  "/etc/awslogs/awslogs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [general]
      state_file = /var/lib/awslogs/agent-state
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      log_stream_name = {instance_id}

  "/etc/awslogs/config/logs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [/var/log/eb-activity.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/eb-activity.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/eb-activity.log
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      multi_line_start_pattern = {datetime_format}

      [/var/log/nodejs/nodejs.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/nodejs/nodejs.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/nodejs/nodejs.log
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      multi_line_start_pattern = {datetime_format}

      [/var/log/nginx/error.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/nginx/error.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/nginx/error.log
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      multi_line_start_pattern = {datetime_format}

      [/var/log/nginx/access.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/nginx/access.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/nginx/access.log
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      multi_line_start_pattern = {datetime_format}

      [/var/log/httpd/error.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/httpd/error.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/httpd/error.log
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      multi_line_start_pattern = {datetime_format}

      [/var/log/httpd/access.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/httpd/access.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/httpd/access.log
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      multi_line_start_pattern = {datetime_format}

commands:
  "01":
    command: chkconfig awslogs on
  "02":
    command: service awslogs restart

在 cloudwatch 代理参考的 FAQ 我看到它说：

如果日志事件的时间戳在未来超过 2 小时，则跳过日志事件

在我的var/log/awslogs.log 中，我确实注意到了这个错误：

someDateTime - cwlogs.push.batch - WARNING - xxxxx - Thread-4 - Skip event: {'timestamp': xxxxxxxxxx, 'start_position': xxxxxxx, 'end_position': xxxxxxxxx}, reason: timestamp is more than 2 hours in future.

如何防止这种情况发生？

【问题讨论】：

标签： node.js amazon-web-services amazon-elastic-beanstalk amazon-cloudwatch amazon-cloudwatchlogs

【解决方案1】：

这是一个迟到的问题，但我在 AWS 支持的帮助下解决了这个问题。我们的负载均衡器被配置为 ping 服务器进行健康检查，这会产生一个特定的日志，比如“在请求中找不到令牌”。健康检查已通过，因为它被配置为检查它，但不断记录相同的字符串导致 CW 代理在为下一个周期创建哈希时无法正常工作。

【讨论】：