几次重启后，kubernetes StatefulSet POD 中的 Microsoft Orleans 崩溃

Question

微软奥尔良 v3.4.3 领事集群 在 K8S 中运行

siloBuilder
     .UseConsulClustering(opt =>
     {
         opt.Address = new Uri(AppConfig.Orleans.ConsulUrl);
         opt.AclClientToken = AppConfig.Orleans.AclClientToken;
     })
     .Configure<ClusterOptions>(options =>
     {
         options.ClusterId = AppConfig.Orleans.ClusterID;
         options.ServiceId = AppConfig.Orleans.ServiceID;
     })
     .siloBuilder.UseKubernetesHosting();

我根据doc为我的POD配置了标签和环境变量。

          - name: ORLEANS_SERVICE_ID #Required by Orleans 
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['orleans/serviceId']
          - name: ORLEANS_CLUSTER_ID #Required by Orleans 
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['orleans/clusterId']
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['statefulset.kubernetes.io/pod-name']
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP

它是一个 StatefulSet，只有 1 个 POD 用于测试。 在初次启动时，它运行良好。 但是，每次我重新启动 POD 时，都会在 Consul 中创建一个新条目。

并且它在随后的启动中崩溃。

日志说

System.AggregateException: One or more errors occurred. (Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Failed to get response from: [S10.18.123.218:11111:361110184])
 ---> Orleans.Runtime.MembershipService.OrleansClusterConnectivityCheckFailedException: Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Failed to get response from: [S10.18.123.218:11111:361110184]
   at Orleans.Runtime.MembershipService.MembershipAgent.ValidateInitialConnectivity()
   at Orleans.Runtime.MembershipService.MembershipAgent.BecomeActive()
   at Orleans.Runtime.MembershipService.MembershipAgent.<>c__DisplayClass26_0.<<Orleans-ILifecycleParticipant<Orleans-Runtime-ISiloLifecycle>-Participate>g__OnBecomeActiveStart|6>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)
   at Orleans.LifecycleSubject.OnStart(CancellationToken ct)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloHost.StartAsync(CancellationToken cancellationToken)
   at Orleans.Hosting.SiloHostedService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at UBS.OrleansServer.EntryPoint.Start() in /app/UBS/OrleansServer/EntryPoint.cs:line 102
   --- End of inner exception stack trace ---

我必须删除 Consul 中的所有条目，然后重新启动 POD，然后一切正常。

POD_NAME对于StatefulSet的POD来说是一样的，每次POD重启都会在Consul中创建一个新条目是否正确？

可能是什么原因？

提前致谢

---

更新 经过几轮崩溃并重新启动，终于不再崩溃了。在日志中我看到以下消息

ProcessTableUpdate (called from DeclareDead) membership table: 5 silos, 1 are Active, 4 are Dead, Version=<31, 28123>. All silos: [SiloAddress=S10.18.123.244:11111:361163684 SiloName=ubs-job-dev-0 Status=Active, SiloAddress=S10.18.123.200:11111:361158057 SiloName=ubs-job-dev-0 Status=Dead, SiloAddress=S10.18.123.210:11111:361161905 SiloName=ubs-job-dev-0 Status=Dead, SiloAddress=S10.18.123.217:11111:361157424 SiloName=ubs-job-dev-0 Status=Dead, SiloAddress=S10.18.123.244:11111:361163558 SiloName=ubs-job-dev-0 Status=Dead]

SiloName 永远不会改变，StatefulSet 中只有一个 POD，但它看到 5 个 silo，其中 4 个已经死亡。似乎每个新的 POD，即使 pod 名称没有改变，也被视为一个新的筒仓。这是预期的吗？

Answer 1

(Failed to get ping responses from 1 of 1 active silos. 
Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. 
Successfully contacted: []. Failed to get response from: [S10.18.123.218:11111:361110184])

看起来您的会员表（在领事中）认为您已经在其中有活动的孤岛。当您的“新”筒仓出现并查看会员表时，它会在表的 IP 地址中看到这些 active 筒仓。

为了保持集群正确，新的joining silo 必须能够与现有的silo 通信。但是，如果成员表不正确（状态为 3/活动的 IP 地址），那么您会遇到问题，即新筒仓尝试 ping active 筒仓并且无法访问它们将无法到达 join 并自行快速.

您有几个解决方案：

• 部署解决方案时清除 consul 表
• 在每次部署时更改部署 ID。

你显然找到了第一个解决方案（清空表格）

见silo lifecycle