在负载测试下访问 MongoDB 时出现连接尝试失败错误

Question

我们开始在我们的应用程序中利用 MongoDB 的搜索功能，以提高我们过去使用 SQL Server 进行搜索的性能。单个测试工作正常，但是现在我们已经开始对 40 个并发“用户”同时执行对 MongoDB 的搜索进行负载测试。在每个负载测试的中途，许多调用开始出错：

MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at MongoDB.Driver.Core.Misc.StreamExtensionMethods.ReadBytes(Stream stream, Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer()
   --- End of inner exception stack trace ---
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer()
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer(Int32 responseTo, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquiredConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocol[TResult](IWireProtocol`1 protocol, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.Command[TResult](ICoreSession session, ReadPreference readPreference, DatabaseNamespace databaseNamespace, BsonDocument command, IEnumerable`1 commandPayloads, IElementNameValidator commandValidator, BsonDocument additionalOptions, Action`1 postWriteAction, CommandResponseHandling responseHandling, IBsonSerializer`1 resultSerializer, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocol(IChannelHandle channel, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocol(IChannelSource channelSource, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.Execute(IReadBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.AggregateOperation`1.Execute(IReadBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteReadOperation[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperation[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperation[TResult](IClientSessionHandle session, IReadOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.Aggregate[TResult](IClientSessionHandle session, PipelineDefinition`2 pipeline, AggregateOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.<>c__DisplayClass19_0`1.<Aggregate>b__0(IClientSessionHandle session)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSession[TResult](Func`2 func, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.Aggregate[TResult](PipelineDefinition`2 pipeline, AggregateOptions options, CancellationToken cancellationToken)
   at RealtorsPropertyResource.Rpr.MongoSearches.Model.QueryPropertiesForKeys[T](PerformanceMonitor performanceMonitor, PropertiesSearchCriteria criteria, RequiredDataResult requiredData)
   at RealtorsPropertyResource.Rpr.MongoSearches.Model.SearchWithMongo(PropertiesSearchCriteria criteria, PerformanceMonitor performanceMonitor)
   at RealtorsPropertyResource.Rpr.MongoSearches.Model.SearchForProperties(PropertiesSearchCriteria criteria, PerformanceMonitor performanceMonitor, Nullable`1 useMongo)
   at RealtorsPropertyResource.Rpr.Websites.Main.AjaxService.ExecuteLoadTestSearch(AppPropertyMode appPropertyMode, SearchType searchType, String query) in C:\BuildAgents\RPR-Website\_work\4\s\Websites\Main\services\AjaxService.Search.cs:line 260
   at SyncInvokeExecuteLoadTestSearch(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

我们针对两个 Mongo 实例进行了测试，一个在 Windows 上运行，一个在 Linux 上运行，两者都遇到了这个错误。成功完成的搜索并不慢，所以我不确定这是一个超时问题，但也许是最大连接问题？不过我很困惑，因为我认为 MongoDB C# 驱动程序的默认连接限制是 100 个连接，而 MongoDB 服务器本身的限制要高得多。

关于如何解决这个仅在 Mongo 处于负载状态时发生的错误的任何建议？

如果相关，这里是我实例化单例 MongoClient 和 IMongoDatabase 的方式：

public static class MongoContext
    {
        private static bool initialized = false;
        private static string connectionString { get; set; }
        private static IMongoDatabase database;

        public static IMongoDatabase Database
        {
            get
            {
                if (database == null)
                    throw new InvalidOperationException("You must initialize the MongoContext first");
                return database;
            }
        }

        public static void Initialize()
        {
            if (!initialized)
            {
                initialized = true;
                connectionString = RprConfigurationManager.GetConnectionString("MongoSearch");
                Console.WriteLine("Connecting to MongoDB search database: " + connectionString);
                var useSsl = connectionString.IndexOf("ssl=true", StringComparison.InvariantCultureIgnoreCase) >= 0;

                var url = new MongoUrl(connectionString);
                var clientSettings = MongoClientSettings.FromUrl(url);
                clientSettings.SslSettings = new SslSettings();
                clientSettings.SslSettings.CheckCertificateRevocation = false;
                clientSettings.UseSsl = useSsl;
                clientSettings.VerifySslCertificate = false;

                var client = new MongoClient(clientSettings);
                var lastSegment = connectionString.Split('/').Last();
                var databaseSegments = lastSegment.Split('?');
                var databaseName = databaseSegments.First();
                databaseName = !String.IsNullOrEmpty(databaseName) ? databaseName : "rpr";
                Console.WriteLine("Connecting to database: " + databaseName);
                database = client.GetDatabase(databaseName);
                SchemaInitializer.RegisterClassMaps();
            }
        }
    }

这里有一些 mongod 日志，以防有帮助：

IXSCAN { CityPlaceId: 1, ForSaleGroupId: 1, DistressedGroupId: 1, OffMarketGroupId：1，ForLeaseGroupId：1，ForSaleDistressedGroupId：1， OffMarketDistressedGroupId：1，OrgId：1，OrgSecurableId：1， OffMarketDate: 1, IsListingOnly: 1 } keysExamined:397707 docsExamined:332738 hasSortStage:1 cursorExhausted:1 numYields:20750 nreturned:1 reslen:1454 锁：{ Global: { acquireCount: { r: 21121 } }，数据库：{acquireCount：{r：21121}}，集合：{ 获取计数：{ r：21121 } } } 存储：{ 数据：{ 字节读取： 7534883557，timeReadingMicros：325960940 } } 协议：op_msg 356835ms 2019-08-19T19:43:08.856+0000 I NETWORK [conn285] 发送错误 对客户端的响应：SocketException：已建立的连接 被主机中的软件中止.. 结束连接 172.27.129.209:65121（连接 ID：285）2019-08-19T19:43:08.856+0000 I NETWORK [conn285] 端连接 172.27.129.209:65121（30 连接现在打开）

Answer 1

提到Here 可能是驱动程序的问题，在我们的例子中，我们将驱动程序版本从 2.5 更改为 2.10 并正常工作。