【问题标题】:Unable to flush task queue within 1200 seconds无法在 1200 秒内刷新任务队列
【发布时间】:2019-12-07 20:07:04
【问题描述】:

在运行我的 automl 管道时,我在 MetricsAndSaveModel 活动期间不断收到错误,这导致我的模型训练运行失败:

2019-12-06 22:48:01,233 - INFO - 295 : ActivityCompleted: Activity=MetricsAndSaveModel, HowEnded=Failure, Duration=1200977.92[ms]
2019-12-06 22:48:01,235 - CRITICAL - 295 : Type: Unclassified
Class: AzureMLException
Message: AzureMLException:
    Message: Failed to flush task queue within 1200 seconds
    InnerException None
    ErrorResponse 
{
    "error": {
        "message": "Failed to flush task queue within 1200 seconds"
    }
}
Traceback:
  File "fit_pipeline.py", line 222, in fit_pipeline
    automl_run_context.batch_save_artifacts(strs_to_save, models_to_upload)
  File "automl_run_context.py", line 201, in batch_save_artifacts
    timeout_seconds=ARTIFACT_UPLOAD_TIMEOUT_SECONDS)
  File "run.py", line 49, in wrapped
    return func(self, *args, **kwargs)
  File "run.py", line 1824, in upload_files
    timeout_seconds=timeout_seconds)
  File "artifacts_client.py", line 167, in upload_files
    results.append(task)
  File "task_queue.py", line 53, in __exit__
    self.flush(self.identity)
  File "task_queue.py", line 126, in flush
    raise AzureMLException("Failed to flush task queue within {} seconds".format(timeout_seconds))

【问题讨论】:

    标签: python azure automl azure-machine-learning-service polynote


    【解决方案1】:

    AutoML 服务中当前的超时限制设置为 20 分钟,我们的产品团队正在努力在未来的版本中将此设置为可配置设置。目前,要增加此限制,您可以修改脚本 automl_run_context.py 以将 ARTIFACT_UPLOAD_TIMEOUT_SECONDS 更新为更高的值,然后重试运行管道。

    【讨论】:

      猜你喜欢
      • 2022-06-10
      • 1970-01-01
      • 1970-01-01
      • 2015-07-03
      • 2020-08-24
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      相关资源
      最近更新 更多