Kubernetes 1.7 calico 日志消息 calico-node -felix-live -bird-live] 和超时 1 (s)

Question

我正在运行 Kubernetes 1.17 集群，Calico CNI 在 containerd 上运行。我的服务器正在运行 CentOS 8，我的所有节点都收到此日志消息。如何停止这些消息？

Mar  8 21:15:48 node03 systemd[1]: Removed slice libcontainer_14289_systemd_test_default.slice.
Mar  8 21:15:48 node03 containerd[1400]: time="2020-03-08T21:15:48.641993033-04:00" level=info msg="ExecSync for "a5e553ee2fd117b0724f56fd9606031f187f4ca0aa468c31f90690f305f49a89" returns with exit code 0"
Mar  8 21:15:50 node03 containerd[1400]: time="2020-03-08T21:15:50.606510411-04:00" level=info msg="ExecSync for "a5e553ee2fd117b0724f56fd9606031f187f4ca0aa468c31f90690f305f49a89" with command [/bin/calico-node -felix-live -bird-live] and timeout 1 (s)"
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14399-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14399-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: Created slice libcontainer_14399_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: Removed slice libcontainer_14399_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14405-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14405-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: Created slice libcontainer_14405_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: Removed slice libcontainer_14405_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14411-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14411-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: Created slice libcontainer_14411_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: Removed slice libcontainer_14411_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14417-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14417-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: Created slice libcontainer_14417_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: Removed slice libcontainer_14417_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14437-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14437-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: Created slice libcontainer_14437_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: Removed slice libcontainer_14437_systemd_test_default.slice.
Mar  8 21:15:50 node03 containerd[1400]: time="2020-03-08T21:15:50.775530563-04:00" level=info msg="Exec process "1f990c60ac0163d4a69c3ac6d2c1c2c53458d872e3d7e950cb89bcf2365c4601" exits with exit code 0 and error <nil>"
Mar  8 21:15:50 node03 containerd[1400]: time="2020-03-08T21:15:50.775615176-04:00" level=info msg="Finish piping "stdout" of container exec "1f990c60ac0163d4a69c3ac6d2c1c2c53458d872e3d7e950cb89bcf2365c4601""

Answer 1

Github 上已经开启 issue，更多详情请见here。

此问题与cgroup-driver 有关。在最新的 Kubernetes 版本中，设置为 systemd 而不是 cgroupfs。

一种解决方法是配置rsyslog。步骤在this site 上有很好的描述。以下短版：

$ cd /etc/rsyslog.d

创建一个包含内容的文件ignore-systemd-session-slice.conf：

if ($programname == "systemd") and ($msg contains "_systemd_test_default.slice" or $msg contains "systemd-test-default-dependencies.scope") then {
  stop
}

之后必须重新启动服务 rsyslog。

$ service rsyslog restart

另一种方法是将cgroupfs 用作cgroup-driver。 这也在this Github thread 上进行了讨论。

---

添加这些以忽略日志

cat /etc/rsyslog.d/ignore-systemd-session-slice.conf文件

if ($programname == "systemd") and ($msg contains "_systemd_test_default.slice" or $msg contains "systemd-test-default-dependencies.scope" or $msg contains " Failed to get system container stats for") then {
  stop
}
if ($programname == "kubelet") and ($msg contains " Failed to get system container stats for" or $msg contains "test_default.slice") then {
  stop
}

if ($programname == "systemd") and ($msg contains "Starting Session" or $msg contains "Started Session" or $msg contains "Created slice" or $msg contains "Starting user-" or $msg contains "Starting User Slice of" or $msg contains "Removed session" or $msg contains "Removed slice User Slice of" or $msg contains "Stopping User Slice of") then {
        stop
}
if ($programname == "containerd") and ($msg regex "Exec process.*with exit code 0" or $msg regex "ExecSync for.*with exit code 0" or $msg contains "Finish piping") then {
  stop
}