停止锁定的按需/触发 Azure Webjob答案

【问题标题】：Stopping Locked On-Demand/Triggered Azure Webjob停止锁定的按需/触发 Azure Webjob
【发布时间】：2016-04-18 02:05:40
【问题描述】：

我在 Azure 中有一个“已触发”的 Web 作业卡在“正在运行” 已经这个状态两天了，计划每5分钟运行一次。

此 Webjob 以前可以正常运行。这个问题似乎发生在星期六（我的时区），这意味着它坏了时没有人在玩它。

到目前为止我所做的尝试：

重新启动 Webjob 附加到的站点。
从 Azure 门户手动运行作业*
重新部署网站
在应用设置中设置 WEBJOBS_STOPPED=1
将计划时间从 5 分钟更改为 6 分钟

*这给了我一个错误：

Could not run job: 'EnrolmentProcessor'.
Please try again. If the problem persists, contact support.

所有这些都没有影响进程，它仍然停留在“运行”状态。

我查看了 KUDU 调试控制台中的文件夹并注意到以下内容：

失败的进程有这些日志行：

[04/15/2016 22:20:02 > 1e4ce4: SYS INFO] Status changed to Initializing
[04/15/2016 22:20:02 > 1e4ce4: SYS INFO] Run script 'TERACC.WebJob.EnrolmentProcessor.exe' with script host - 'WindowsScriptHost'
[04/15/2016 22:20:02 > 1e4ce4: SYS INFO] Status changed to Running

上一个成功的进程有这些日志行：

[04/15/2016 22:15:01 > 1e4ce4: SYS INFO] Status changed to Initializing
[04/15/2016 22:15:01 > 1e4ce4: SYS INFO] Run script 'TERACC.WebJob.EnrolmentProcessor.exe' with script host - 'WindowsScriptHost'
[04/15/2016 22:15:01 > 1e4ce4: SYS INFO] Status changed to Running
[04/15/2016 22:18:51 > 1e4ce4: SYS INFO] Status changed to Success

有一个名为 triggeredJob.lock 的文件在失败时具有以下 StackTrace：

2016-04-15T22:20:02   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at Kudu.Core.Infrastructure.LockFile.WriteLockInfo(Stream lockStream)
   at Kudu.Core.Infrastructure.LockFile.Lock()
   at Kudu.Core.Jobs.TriggeredJobRunner.StartJobRun(TriggeredJob triggeredJob, JobSettings jobSettings, String trigger, Action`2 reportAction)
   at Kudu.Core.Jobs.TriggeredJobsManager.InvokeTriggeredJob(String jobName, String arguments, String trigger)
   at Kudu.Services.Jobs.JobsController.InvokeTriggeredJob(String jobName, String arguments)
   at lambda_method(Closure , Object , Object[] )
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass10.<GetExecutor>b__9(Object instance, Object[] methodParameters)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)
   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Web.Http.Controllers.ApiControllerActionInvoker.InvokeActionAsyncCore(HttpActionContext actionContext, CancellationToken cancellationToken)
   at System.Web.Http.Controllers.ApiControllerActionInvoker.InvokeActionAsync(HttpActionContext actionContext, CancellationToken cancellationToken)
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>b__0(ActionInvoker innerInvoker)
   at System.Web.Http.Controllers.ActionFilterResult.<>c__DisplayClass10`1.<InvokeActionWithActionFilters>b__f()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Web.Http.Filters.ActionFilterAttribute.CallOnActionExecutedAsync(HttpActionContext actionContext, CancellationToken cancellationToken, Func`1 continuation)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__0.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Web.Http.Filters.ActionFilterAttribute.ExecuteActionFilterAsyncCore(HttpActionContext actionContext, CancellationToken cancellationToken, Func`1 continuation)
   at System.Web.Http.Filters.ActionFilterAttribute.System.Web.Http.Filters.IActionFilter.ExecuteActionFilterAsync(HttpActionContext actionContext, CancellationToken cancellationToken, Func`1 continuation)
   at System.Web.Http.Controllers.ActionFilterResult.<>c__DisplayClassb.<>c__DisplayClassd.<InvokeActionWithActionFilters>b__9()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Web.Http.Filters.ActionFilterAttribute.CallOnActionExecutedAsync(HttpActionContext actionContext, CancellationToken cancellationToken, Func`1 continuation)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__0.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Web.Http.Filters.ActionFilterAttribute.ExecuteActionFilterAsyncCore(HttpActionContext actionContext, CancellationToken cancellationToken, Func`1 continuation)
   at System.Web.Http.Filters.ActionFilterAttribute.System.Web.Http.Filters.IActionFilter.ExecuteActionFilterAsync(HttpActionContext actionContext, CancellationToken cancellationToken, Func`1 continuation)
   at System.Web.Http.Controllers.ActionFilterResult.<>c__DisplayClassb.<>c__DisplayClassd.<InvokeActionWithActionFilters>b__9()
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Web.Http.Controllers.ActionFilterResult.ExecuteAsync(CancellationToken cancellationToken)
   at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Web.Http.Controllers.ExceptionFilterResult.ExecuteAsync(CancellationToken cancellationToken)
   at System.Web.Http.ApiController.ExecuteAsync(HttpControllerContext controllerContext, CancellationToken cancellationToken)
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageInvoker.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Web.Http.Dispatcher.HttpRoutingDispatcher.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Web.Http.HttpServer.<SendAsync>d__0.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Web.Http.HttpServer.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageInvoker.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Web.Http.WebHost.HttpControllerHandler.<ProcessRequestAsyncCore>d__0.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Web.Http.WebHost.HttpControllerHandler.ProcessRequestAsyncCore(HttpContextBase contextBase)
   at System.Web.Http.WebHost.HttpControllerHandler.ProcessRequestAsync(HttpContext context)
   at System.Web.HttpTaskAsyncHandler.<>c__DisplayClass4_0.<System.Web.IHttpAsyncHandler.BeginProcessRequest>b__0()
   at System.Web.TaskAsyncHelper.BeginTask(Func`1 taskFunc, AsyncCallback callback, Object state)
   at System.Web.HttpTaskAsyncHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
   at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

这向我表明，在尝试处理 Webjob 并使其处于这种损坏状态时出现问题。

我不担心在“运行”的过程中丢失数据，我只需要让 Webjob 再次正常运行。我该怎么做才能中止它卡住的那个并让它重新启动？

【问题讨论】：

您能直接或indirectly 分享您的网络应用名称吗？这将有助于调查。谢谢！
@DavidEbbo - 当然可以。这是同一订阅的网站：testforfurtherinfo.azurewebsites.net。带有错误 webjob 的站点以 test（不是 testapi）结束。卡住的 Webjob 称为 EnrolmentProcessor
好像有死锁。你能告诉我们你的代码吗？

标签： azure azure-webjobs

【解决方案1】：

它似乎进入了一个糟糕的状态。我认为关键是删除那个triggeredJob.lock 文件。问题是它被锁定并且不想被删除。

请尝试以下步骤，看看是否允许删除：

使用这些special steps 停止站点和 SCM 站点
使用 FTP 连接并删除锁定文件
重新启动站点（例如从门户）

如果这不起作用，我们将尝试其他方法。我想了解根本原因，但让我们先尝试让您运行。

【讨论】：

嗨@DavidEbbo，感谢您迄今为止的帮助。不幸的是，这仍然不起作用，当尝试删除锁定文件时，我仍然收到以下错误：“550 进程无法访问该文件，因为它正在被另一个进程使用。”
我进入虚拟机并能够关闭句柄，然后重命名文件（我将其保留为 triggeredJob.lock.old 以作记录）。但令我惊讶的是，关闭 SCM 站点并没有奏效，因为锁由 SCM 站点的 w3wp 持有。为确保您使用了我提供的链接中的技术并更改了state 和scmSiteAlsoStopped?无论如何，希望事情现在恢复到良好的状态。请验证。
我注意到的一件事是，这个 WebJob 似乎需要 7 分钟或更长时间才能运行，但安排得比这更频繁。这将导致某些调度程序运行失败并出现 409 CONFLICT，然后重试。这可能是无害的，但最好少安排一些时间。
嗨@DavidEbbo，你的修复效果很好！谢谢！我确实确保状态设置为“已停止”并且 scmSiteAlsoStopped 设置为“真”。我检查并在更新这些设置后无法浏览到 {site}.scm.azurewebsites.net 和实际站点。然后我进入并尝试更改再次给出 550 错误的锁定文件。好的，很酷，我会将时间表增加到 10 分钟，以便在再次触发之前完成作业。再次感谢您的帮助。
很高兴这有效，尽管它留下了一些未解之谜。如果再次发生这种情况，我们将尝试进一步了解。