AppFabric 扩展问题故障排除（间歇性 ErrorCode<ERRCA0017>:SubStatus<ES0006> 错误）

Question

我们已经为我们的 Web 应用程序实施了 AppFabric Windows Server 缓存。最初，我们能够毫无问题地使用缓存。然后我们将流量增加了大约 100 倍，并开始遇到间歇性异常。异常大约每 2 天发生一次，每次大约一分钟。

我们的配置：

• 9 个 Web 服务器在缓存中插入/检索对象：
  • 主要是临时的 500 字节操作类型对象
  • 使用 1 个命名区域
  • 使用标签存储的对象
  • 批量检索给定标签
• 缓存集群：
  • 1 台主机（领导）AppFabric 1.1（get-cachehost 报告的版本为 3）
  • SQL 配置提供程序
  • 主机上有 96GB 的 RAM，默认 50% (48GB) 分配给 AppFabric
  • 缓存主机Config
  • 缓存客户端Config

错误发生的顺序（例外情况发生在九个网络服务器中的每一个在一分钟内）：

• System.Net.Sockets.SocketException : 现有连接被远程主机强行关闭 Microsoft.ApplicationServer.Caching.DataCacheException：ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown. ---> System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:15:00'. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host --- End of inner exception stack trace --- at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result) at System.ServiceModel.Channels.FramingDuplexSessionChannel.EndReceive(IAsyncResult result) at Microsoft.ApplicationServer.Caching.WcfClientChannel.CompleteProcessing(IAsyncResult result) --- End of inner exception stack trace --- at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody) at Microsoft.ApplicationServer.Caching.DataCache.GetNextBatch(String region, DataCacheTag[] tags, GetByTagsOperation op, IMonitoringListener listener, Byte[][]& state, Boolean& more) at Microsoft.ApplicationServer.Caching.CacheEnumerator.MoveNext() at System.Linq.Enumerable.WhereSelectEnumerableIterator'2.MoveNext() at System.Linq.Enumerable.<ExceptIterator>d__99'1.MoveNext() at System.Collections.Generic.List'1..ctor(IEnumerable'1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable'1 source)

• Microsoft.ApplicationServer.Caching.DataCacheException： ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.) at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody) at Microsoft.ApplicationServer.Caching.DataCache.GetNextBatch(String region, DataCacheTag[] tags, GetByTagsOperation op, IMonitoringListener listener, Byte[][]& state, Boolean& more) at Microsoft.ApplicationServer.Caching.CacheEnumerator.MoveNext() at System.Linq.Enumerable.WhereSelectEnumerableIterator'2.MoveNext() at System.Linq.Enumerable.<ExceptIterator>d__99'1.MoveNext() at System.Collections.Generic.List'1..ctor(IEnumerable'1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable'1 source)

• Microsoft.ApplicationServer.Caching.DataCacheException： ErrorCode<ERRCA0018>:SubStatus<ES0001>:The request timed out. at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody) at Microsoft.ApplicationServer.Caching.DataCache.GetNextBatch(String region, DataCacheTag[] tags, GetByTagsOperation op, IMonitoringListener listener, Byte[][]& state, Boolean& more) at Microsoft.ApplicationServer.Caching.CacheEnumerator.MoveNext() at System.Linq.Enumerable.WhereSelectEnumerableIterator'2.MoveNext() at System.Linq.Enumerable.<ExceptIterator>d__99'1.MoveNext() at System.Collections.Generic.List'1..ctor(IEnumerable'1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable'1 source)

我们还在缓存服务器上创建了一个跟踪日志会话，以捕获更多信息以诊断问题 - 任何有关如何分析此问题的建议都将不胜感激（如果需要，我可以提供）。

我们还监控了各种 AppFabric、CLR 和网络性能计数器，下面是事件发生时的屏幕截图：

在此先感谢您分享解决此问题的任何想法或建议。

更新 1

以下是在间歇性错误期间 AppFabric 缓存服务器上连续发生的异常（从跟踪日志中提取）：

• System.ServiceModel.CommunicationException: The socket connection was aborted because an asynchronous send to the socket did not complete within the allotted timeout of 00:00:00.0082078. The time allotted to this operation may have been a portion of a longer timeout. ---> System.ObjectDisposedException: The socket connection has been disposed. Object name: 'System.ServiceModel.Channels.SocketConnection'. --- End of inner exception stack trace --- at System.ServiceModel.Channels.SocketConnection.ThrowIfNotOpen() at System.ServiceModel.Channels.SocketConnection.BeginRead(Int32 offset, Int32 size, TimeSpan timeout, WaitCallback callback, Object state) at System.ServiceModel.Channels.SessionConnectionReader.BeginReceive(TimeSpan timeout, WaitCallback callback, Object state) at System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.PerformOperation(TimeSpan timeout) at System.ServiceModel.Channels.SynchronizedMessageSource.SynchronizedAsyncResult'1..ctor(SynchronizedMessageSource syncSource, TimeSpan timeout, AsyncCallback callback, Object state) at System.ServiceModel.Channels.FramingDuplexSessionChannel.BeginReceive(TimeSpan timeout, AsyncCallback callback, Object state) at Microsoft.ApplicationServer.Caching.WcfServerChannel.CompleteProcessing(IAsyncResult result)

• System.ServiceModel.CommunicationObjectAbortedException: The communication object, System.ServiceModel.Channels.ServerSessionPreambleConnectionReader+ServerFramingDuplexSessionChannel, cannot be used for communication because it has been Aborted. at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result) at System.ServiceModel.Channels.FramingDuplexSessionChannel.OnEndSend(IAsyncResult result) at Microsoft.ApplicationServer.Caching.ReplyContext.EndSend(IAsyncResult result)

• System.ServiceModel.CommunicationObjectFaultedException: The communication object, System.ServiceModel.Channels.ServerSessionPreambleConnectionReader+ServerFramingDuplexSessionChannel, cannot be used for communication because it is in the Faulted state. at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen() at System.ServiceModel.Channels.OutputChannel.Send(Message message, TimeSpan timeout) at Microsoft.ApplicationServer.Caching.ReplyContext.Reply(Message message, TimeSpan timeout)

• System.TimeoutException: Sending to via http://www.w3.org/2005/08/addressing/anonymous timed out after 00:00:15. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: Cannot claim lock within the allotted timeout of 00:00:15. The time allotted to this operation may have been a portion of a longer timeout. --- End of inner exception stack trace --- at System.ServiceModel.Channels.FramingDuplexSessionChannel.OnSend(Message message, TimeSpan timeout) at System.ServiceModel.Channels.OutputChannel.Send(Message message, TimeSpan timeout) at Microsoft.ApplicationServer.Caching.ReplyContext.Reply(Message message, TimeSpan timeout)

更新 2

经过一天的故障排除后，我们采取了以下措施并取得了一些改进：

1. 基于this 和this，我们将maxConnectionsToServer 增加到3。结果，AppFabric Caching:Cache perf 计数器记录的每秒客户端请求数增加了 50%，但间歇性错误并未停止发生

2. 我们在 缓存服务器 配置上将 maxBufferSize 和 maxBufferPoolSize 增加到 2147483647 (int32.max)。到目前为止，我们能够处理没有错误的 300 倍流量。我们将继续增加流量和监控。更多更新敬请关注

更新 3

我们向集群添加了另外两台主机，每台 16GB，并启用了 HighAvailability 模式（通过Secondaries=1）。目前，原始主机保留在集群中，内存为 96GB - 所有主机都有cacheSize = 12GB。在缓存客户端上，我们将MaxConnectionToServer 增加到12（每个核心1 个）。以下是我们的发现：

• 有时我们会得到（每 10 分钟一次或两次）：
  • ErrorCode<ERRCA0017>:SubStatus<ES0005>:There is a temporary failure. Please retry later. (There was a contention on the store.)
  • ErrorCode<ERRCA0017>:SubStatus<ES0004>:There is a temporary failure. Please retry later. (Replication queue was full. This may happen during reconfiguration of cluster hosts.)
• 原来的 96GB 缓存主机仍然会出现 1 分钟的中断，如上所述。新的缓存主机没有遇到中断

我们计划从原始缓存主机中移除 80GB 内存。后续更新。

更新 4

问题似乎已通过将缓存主机中的 RAM 量减少到 16GB 来解决。我们不再看到流量增加到 400 倍的间歇性错误。好像封箱了。现在进入下一期：High Availability

Answer 1

1. 你安装http://support.microsoft.com/kb/983182和http://support.microsoft.com/kb/2527387了吗？

2. 您是否在您的代码中检查异常和 retrylater bool？

                   catch (DataCacheException ex2)
               {
                   if (ex2.ErrorCode == DataCacheErrorCode.RetryLater)
                   {
   

   使用命名区域会强制服务器将该命名区域的值推送到单个服务器，而不是将哈希值分散到所有缓存服务器中。 （“为了提供这种添加的搜索功能，区域中的对象仅限于单个缓存主机。”http://msdn.microsoft.com/en-us/library/ee790985(v=azure.10).aspx）

我建议您将命名区域分片到另外 2 台服务器上，然后将它们放在一个集群中。通过这种方式，您可以在运行 GC 并尝试找到更多内存来放置和存储对象和标签时将异常限制在较小的服务器上。

Answer 2

将 Jeff-ITGuy 的答案转发到social.msdn.microsoft.com：

您遇到的问题似乎与我目前与 Microsoft 合作的问题几乎相同。如果是同样的问题，很可能是GC耗时过长，导致AppFabric响应时间延迟。从您的性能计数器来看，当您开始遇到问题时，GC 时间似乎突然增加，因此可能是同一个问题。

Microsoft 正在积极调查此问题。同时，为了缓解这个问题（至少从我们的发现来看），您可以使用更少的内存运行更多的服务器（缩小 GC 正在处理的内存空间的大小），并且您可以增加客户端上的 RequestTimeout。默认设置为 15000（15 秒），但我们尝试将其提高到 30000，这有助于消除一些问题。在我看来，这不是一个好的长期解决方案，只是传递信息。我已经看到只有 24gb 内存（12gb 用于缓存）的服务器的问题，只有当我们尝试将 4gb 设置为缓存的 8gb 服务器时，它才真正明显好转——这导致 GC 做得更好。

希望对您有所帮助，但如果这是问题所在，我认为还没有解决方案。

确实有帮助，在我们将缓存主机 RAM 减少到 16GB 后，间歇性错误停止了。

Answer 3

目前可在此处找到此问题的修复程序： http://support.microsoft.com/kb/2787717