在自动 ML 上无法在 120 秒内刷新任务队列

Question

我在 Azure 上运行自动 ML，但它失败了，这是子作业给我的错误。

Starting the automl_batch_driver setup...
Set enable_streaming flag to False
Batch Run Id in the real script:  AutoML_11f87010-f7b0-4819-aecb-ad6b31f2469a_worker_11
2022-06-07 17:46:03,026817 - INFO - Beginning batch driver wrapper.
2022-06-07 17:46:03.297 - INFO - Successfully got the cache data store, caching enabled.
2022-06-07 17:46:03.297 - INFO - Took 0.13631367683410645 seconds to retrieve cache data store
2022-06-07 17:46:03.306 - INFO - No files are available for the cache store locally, downloading files from the Run.
2022-06-07 17:48:04.348 - ERROR - Type: {'code': 'ResourceExhausted', 'inner_error': {'code': 'Timeout'}}
Class: AzureMLException
Message: AzureMLException:
    Message: Failed to flush task queue within 120 seconds
    InnerException None
    ErrorResponse 
{
    "error": {
        "code": "UserError",
        "message": "Failed to flush task queue within 120 seconds",
        "inner_error": {
            "code": "ResourceExhausted",
            "inner_error": {
                "code": "Timeout"
            }
        }
    }
}

Answer 1

要解决此Failed to flush task queue within 120 seconds 错误，请尝试以下方法：

• 按照harneetvirk 的建议，尝试使用run.download_file 而不是run.download_files

对于example：

import os

   run = experiment.start_logging()
   file_name_1 = 'important_file_1'
   file_name_2 = 'important_file_2'
   run.upload_files(names=[file_name_1, file_name_2],
                       paths=['path/on/disk/file_1.txt', 'other/path/on/disk/file_2.txt'])

   run.download_file(file_name_1, 'file_1.txt')

   os.mkdir("path")  # The path must exist
   run.download_file(file_name_2, 'path/file_2.txt')

• 如果您要上传工件，那么按照RohitMungi 的建议，增加automl_run_context.py 的限制以将ARTIFACT_UPLOAD_TIMEOUT_SECONDS 更新为更高的值。