编写 Google Apps 脚本以在 BigQuery“查询执行期间超出资源”时引发错误

Question

我正在使用脚本来运行视图并使用查询结果来更新 BigQuery 中的表。然后，我在 Data Studio 中将这些表用于表/可视化。

我注意到的一个问题是，有时我的视图会突然抛出一个错误“查询执行期间资源超出”。但是，我的 Google Apps 脚本将成功运行，但表格将保持不变，因为无法执行视图。因此，我的 Data Studio 可视化和表格中的数据将是陈旧的。除非我去编辑视图并看到它抛出错误，否则我将无法知道这一点。

是否有任何方法可以在我的脚本中添加一些内容，以便在 Bigquery 抛出“查询执行期间超出资源”错误时脚本将失败或通知我？这样我就不会因为错误而在不知不觉中拥有过时的数据。

  var configuration = {
    "query": {
    "useQueryCache": false,
    "destinationTable": {
          "projectId": "abc-123",
          "datasetId": "report_tables",
          "tableId": "dashboard_1"
        },
    "writeDisposition": "WRITE_TRUNCATE",
    "createDisposition": "CREATE_IF_NEEDED",
    "allowLargeResults": true,
    "useLegacySql": false,
    "query": "SELECT * FROM `abc-123.report_tables.dashboard_1_view`"
    }
  };

  var job = {
    "configuration": configuration
  };

  var jobResult = BigQuery.Jobs.insert(job, "abc-123");
  Logger.log(jobResult);
}

我希望您能帮我在上面的脚本中添加一些子句，以便在 BigQuery 视图中超出资源时，这将无法执行或以某种方式通知我，因此我知道要在 BigQuery 中更正问题.

感谢您的帮助！

Answer 1

您的函数正在成功，因为 BigQuery 上的所有作业/查询都是异步的。 var jobResult = BigQuery.Jobs.insert(job, "abc-123"); 行仅提交作业/查询到 BigQuery 服务以开始执行。然后你的函数返回。

因此，您需要在作业执行时等待并轮询作业的status，然后在作业失败时处理任何错误，如下所示：

  [..]
  var job = BigQuery.Jobs.insert(job, "abc-123");
  Logger.log(job.status.state);
  var jobId = job.jobReference.jobId;

  // Check on status of the Query Job.
  var sleepTimeMs = 500;
  while (job.status.state !== "DONE") {
    Utilities.sleep(sleepTimeMs);
    job = BigQuery.Jobs.get("abc-123", jobId);
    Logger.log(job.status.state);
  }
  if(job.status.errors != null && job.status.errors.length > 0) {
     Logger.log("FAILED:" + job.status.errors);
  } else {
     Logger.log("SUCCEEDED")
  }
  [..]

样本输出（失败）：

[19-06-03 19:54:48:557 AEST] RUNNING
[19-06-03 19:54:49:161 AEST] RUNNING
[19-06-03 19:54:49:789 AEST] RUNNING
[19-06-03 19:54:50:368 AEST] RUNNING
[19-06-03 19:54:51:147 AEST] RUNNING
[19-06-03 19:54:51:783 AEST] RUNNING
[19-06-03 19:54:52:356 AEST] RUNNING
[19-06-03 19:54:52:957 AEST] RUNNING
[19-06-03 19:54:53:564 AEST] RUNNING
[19-06-03 19:54:54:151 AEST] RUNNING
[19-06-03 19:54:54:748 AEST] RUNNING
[19-06-03 19:54:55:338 AEST] RUNNING
[19-06-03 19:54:55:954 AEST] RUNNING
[19-06-03 19:54:56:539 AEST] RUNNING
[19-06-03 19:54:57:107 AEST] RUNNING
[19-06-03 19:54:57:724 AEST] RUNNING
[19-06-03 19:54:58:513 AEST] RUNNING
[19-06-03 19:54:59:524 AEST] RUNNING
[19-06-03 19:55:00:144 AEST] RUNNING
[19-06-03 19:55:00:993 AEST] RUNNING
[19-06-03 19:55:01:613 AEST] RUNNING
[19-06-03 19:55:02:219 AEST] RUNNING
[19-06-03 19:55:02:989 AEST] RUNNING
[19-06-03 19:55:03:557 AEST] RUNNING
[19-06-03 19:55:04:123 AEST] RUNNING
[19-06-03 19:55:04:684 AEST] RUNNING
[19-06-03 19:55:05:408 AEST] RUNNING
[19-06-03 19:55:06:018 AEST] RUNNING
[19-06-03 19:55:06:603 AEST] RUNNING
[19-06-03 19:55:07:215 AEST] RUNNING
[19-06-03 19:55:07:789 AEST] RUNNING
[19-06-03 19:55:08:424 AEST] RUNNING
[19-06-03 19:55:08:985 AEST] RUNNING
[19-06-03 19:55:09:580 AEST] RUNNING
[19-06-03 19:55:10:184 AEST] RUNNING
[19-06-03 19:55:10:802 AEST] RUNNING
[19-06-03 19:55:11:391 AEST] RUNNING
[19-06-03 19:55:11:984 AEST] RUNNING
[19-06-03 19:55:12:564 AEST] RUNNING
[19-06-03 19:55:13:154 AEST] RUNNING
[19-06-03 19:55:13:752 AEST] RUNNING
[19-06-03 19:55:14:372 AEST] RUNNING
[19-06-03 19:55:14:974 AEST] RUNNING
[19-06-03 19:55:15:661 AEST] RUNNING
[19-06-03 19:55:16:227 AEST] RUNNING
[19-06-03 19:55:16:815 AEST] RUNNING
[19-06-03 19:55:17:416 AEST] RUNNING
[19-06-03 19:55:18:002 AEST] DONE
[19-06-03 19:55:18:003 AEST] FAILED:{"reason":"resourcesExceeded","message":"Resources exceeded during query execution: The query could not be executed in the allotted memory. Peak usage: 121% of limit.\nTop memory consumer(s):\n  ORDER BY operations: 99%\n  other/unattributed: 1%\n"}

样本输出（成功）：

[19-06-03 19:59:48:206 AEST] RUNNING
[19-06-03 19:59:48:820 AEST] RUNNING
[19-06-03 19:59:49:433 AEST] RUNNING
[19-06-03 19:59:50:015 AEST] DONE
[19-06-03 19:59:50:016 AEST] SUCCEEDED

需要考虑的是，使用 Google Apps 脚本会将所有这些与您的个人 Google 帐户（即不是服务帐户）联系起来。这对你来说可能没问题，但在企业中这有点禁忌。

最后，我可能会将您的脚本移至 Cloud Build。我认为它更强大、更易于维护、更易于使用和更灵活。