【发布时间】:2021-06-30 16:59:11
【问题描述】:
我们已经公开了一个 JobService(它是使用 kestrel 的 Dotnet Core 服务),它为我们的对象(JobDocument)进行 MongoDB 更新,并且看到在对 createJob 的 Post 调用的情况下,它实际上触发了 MongoDB.Driver.MongoCollectionBase`1.InsertOneAsync有一个性能冲击,即。大约 200 个并发请求使服务无响应。经过调试和使用一些工具,发现问题出在api调用InsertOneAsync。这是使用我们对 GenerateId 的自定义实现,它使用同步函数调用 FindOneAndUpdate。GenerateId 是否有异步方式。如何做到这一点?参考使用指向相同的工具 Ben Blocking 检测器附加堆栈:
{"Timestamp":"2021-04-03T15:22:38.5854816+05:30","Level":"Warning","MessageTemplate":"Blocking method has been invoked and blocked, this can lead to threadpool starvation.\r\n{stackTrace}","Properties":{"stackTrace":"
at System.Threading.Tasks.TplEventSource.TaskWaitBegin(Int32 OriginatingTaskSchedulerID, Int32 OriginatingTaskID, Int32 TaskID, TaskWaitBehavior Behavior, Int32 ContinueWithTaskID)\r\n
at System.Threading.Tasks.Task.InternalWaitCore(Int32 millisecondsTimeout, CancellationToken cancellationToken)\r\n
at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)\r\n
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)\r\n
at MongoDB.Driver.Core.Connections.BinaryConnection.Open(CancellationToken cancellationToken)\r\n
at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.PooledConnection.Open(CancellationToken cancellationToken)\r\n
at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquiredConnection.Open(CancellationToken cancellationToken)\r\n
at MongoDB.Driver.Core.Servers.Server.GetChannel(CancellationToken cancellationToken)\r\n
at MongoDB.Driver.Core.Bindings.ServerChannelSource.GetChannel(CancellationToken cancellationToken)\r\n
at MongoDB.Driver.Core.Bindings.ChannelSourceHandle.GetChannel(CancellationToken cancellationToken)\r\n
at MongoDB.Driver.Core.Operations.RetryableWriteContext.Initialize(CancellationToken cancellationToken)\r\n
at MongoDB.Driver.Core.Operations.RetryableWriteContext.Create(IWriteBinding binding, Boolean retryRequested, CancellationToken cancellationToken)\r\n
at MongoDB.Driver.Core.Operations.RetryableWriteOperationExecutor.Execute[TResult](IRetryableWriteOperation`1 operation, IWriteBinding binding, Boolean retryRequested, CancellationToken cancellationToken)\r\n
at MongoDB.Driver.Core.Operations.FindAndModifyOperationBase`1.Execute(IWriteBinding binding, CancellationToken cancellationToken)\r\n
at MongoDB.Driver.OperationExecutor.ExecuteWriteOperation[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperation[TResult](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.FindOneAndUpdate[TProjection](IClientSessionHandle session, FilterDefinition`1 filter, UpdateDefinition`1 update, FindOneAndUpdateOptions`2 options, CancellationToken cancellationToken)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.<>c__DisplayClass53_0`1.<FindOneAndUpdate>b__0(IClientSessionHandle session)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSession[TResult](Func`2 func, CancellationToken cancellationToken)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.FindOneAndUpdate[TProjection](FilterDefinition`1 filter, UpdateDefinition`1 update, FindOneAndUpdateOptions`2 options, CancellationToken cancellationToken)\r\n
at JobManager.Data.MongoDB.IdGenerator.JobIdGenerator.GenerateId(Object container, Object document)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.AssignId(TDocument document)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.ConvertWriteModelToWriteRequest(WriteModel`1 model, Int32 index)\r\n
at System.Linq.Enumerable.SelectIterator[TSource,TResult](IEnumerable`1 source, Func`3 selector)+MoveNext()\r\n
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)\r\n
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)\r\n
at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation..ctor(CollectionNamespace collectionNamespace, IEnumerable`1 requests, MessageEncoderSettings messageEncoderSettings)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.CreateBulkWriteOperation(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.BulkWriteAsync(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.BulkWriteAsync(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.<>c__DisplayClass25_0.<BulkWriteAsync>b__0(IClientSessionHandle session)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)\r\n
at MongoDB.Driver.MongoCollectionImpl`1.BulkWriteAsync(IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)\r\n
at MongoDB.Driver.MongoCollectionBase`1.<>c__DisplayClass68_0.<InsertOneAsync>b__0(IEnumerable`1 requests, BulkWriteOptions bulkWriteOptions)\r\n
at MongoDB.Driver.MongoCollectionBase`1.InsertOneAsync(TDocument document, InsertOneOptions options, Func`3 bulkWriteAsync)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at MongoDB.Driver.MongoCollectionBase`1.InsertOneAsync(TDocument document, InsertOneOptions options, Func`3 bulkWriteAsync)\r\n
at MongoDB.Driver.MongoCollectionBase`1.InsertOneAsync(TDocument document, InsertOneOptions options, CancellationToken cancellationToken)\r\n
at Code.DataProtection.JobManager.Data.MongoDB.Repositories.MongoJobRepository.Code.DataProtection.JobManager.Data.Contract.IJobRepository.CreateJob(Job job)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Code.DataProtection.JobManager.Data.MongoDB.Repositories.MongoJobRepository.Code.DataProtection.JobManager.Data.Contract.IJobRepository.CreateJob(Job job)\r\n
at Code.DataProtection.JobManager.Services.JobService.CreateJob(Job job)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Code.DataProtection.JobManager.Services.JobService.CreateJob(Job job)\r\n
at Code.DataProtection.JobManager.Services.JobService.Code.DataProtection.JobManager.Contract.Interfaces.IJobService.CreateJobWithWorkflow(Job job)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Code.DataProtection.JobManager.Services.JobService.Code.DataProtection.JobManager.Contract.Interfaces.IJobService.CreateJobWithWorkflow(Job job)\r\n
at Code.DataProtection.JobManager.Api.Controllers.v2.JobsController.CreateJob(Job job)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Code.DataProtection.JobManager.Api.Controllers.v2.JobsController.CreateJob(Job job)\r\n at lambda_method(Closure , Object , Object[] )\r\n
at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\r\n at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAwaitedAsync()\r\n
at Microsoft.AspNetCore.Mvc.Filters.ActionFilterAttribute.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Microsoft.AspNetCore.Mvc.Filters.ActionFilterAttribute.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAwaitedAsync()\r\n
at Code.DataProtection.Common.Utils.Filters.ContextFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Code.DataProtection.Common.Utils.Filters.ContextFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAwaitedAsync()\r\n
at Microsoft.AspNetCore.Mvc.Controller.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)\r\n
at Microsoft.AspNetCore.Mvc.Filters.ControllerActionFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeNextResourceFilter()\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)\r\n
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeAsync()\r\n
at Microsoft.AspNetCore.Mvc.Routing.ActionEndpointFactory.<>c__DisplayClass7_0.<CreateRequestDelegate>b__0(HttpContext context)\r\n
at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)\r\n
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)\r\n
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\r\n
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\r\n
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass4_1.<UseMiddleware>b__2(HttpContext context)\r\n
at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)\r\n
at CorrelationId.CorrelationIdMiddleware.Invoke(HttpContext context, ICorrelationContextFactory correlationContextFactory)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at CorrelationId.CorrelationIdMiddleware.Invoke(HttpContext context, ICorrelationContextFactory correlationContextFactory)\r\n
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass4_1.<UseMiddleware>b__2(HttpContext context)\r\n
at Code.DataProtection.Common.Utils.ExceptionHandler.ExceptionHandlerMiddleware.Invoke(HttpContext context)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Code.DataProtection.Common.Utils.ExceptionHandler.ExceptionHandlerMiddleware.Invoke(HttpContext context)\r\n
at Ben.Diagnostics.BlockingDetectionMiddleware.Invoke(HttpContext httpContext)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Ben.Diagnostics.BlockingDetectionMiddleware.Invoke(HttpContext httpContext)\r\n
at Microsoft.AspNetCore.Mvc.Versioning.ApiVersioningMiddleware.InvokeAsync(HttpContext context)\r\n
at Microsoft.AspNetCore.HostFiltering.HostFilteringMiddleware.Invoke(HttpContext context)\r\n
at Microsoft.AspNetCore.Hosting.HostingApplication.ProcessRequestAsync(Context context)\r\n
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)\r\n
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)\r\n
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)\r\n
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnectionMiddleware`1.OnConnectionAsync(ConnectionContext connectionContext)\r\n
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection.ExecuteAsync()\r\n
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection.ExecuteAsync()\r\n
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection.System.Threading.IThreadPoolWorkItem.Execute()\r\n
at System.Threading.ThreadPoolWorkQueue.Dispatch()\r\n
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()\r\n","EventId":{"Id":6,"Name":"BlockingMethodCalled"},"SourceContext":"Ben.Diagnostics.BlockingMonitor","ActionId":"25c60a2a-f366-415b-8f16-66e10f134608","ActionName":"Code.DataProtection.JobManager.Api.Controllers.v2.JobsController.CreateJob (JobManager.Api)","RequestId":"0HM7MGOIK401C:00000001","RequestPath":"/api/jobmanager/v2.0/jobs","SpanId":"|11136163-4c80e7d34af376d0.","TraceId":"11136163-4c80e7d34af376d0","ParentId":"","ConnectionId":"0HM7MGOIK401C"}}
【问题讨论】:
-
您的自定义 id 生成代码是否使用
FindOneAndUpdate进行顺序 id 生成以增加数据库中的计数器文档?如果没有,将有助于查看该代码。 -
是的,我们正在使用 FindOneAndUpdate 来增加计数器。我们不能使用 FindOneAndUpdateAsync,因为它要求 GenerateId api 也是异步的,但它不受支持。
-
public class JobIdGenerator : IIdGenerator{ public object GenerateId(object container, object doc) { IMongoCollection<SequenceDocument> idSeqColl = (IMongoCollection<JobDocument>)container).Database.GetCollection<SequenceDocument>("Sequence"); var filter = Builders<SequenceDocument>.Filter.Empty; var update = Builders<SequenceDocument>.Update.Inc(a => a.Value, 1); return idSeqColl.FindOneAndUpdate(filter, update, new FindOneAndUpdateOptions<SequenceDocument, SequenceDocument> {ReturnDocument = ReturnDocument.After, IsUpsert = true}).Value;
标签: c# mongodb .net-core mongodb-.net-driver kestrel