使用 BigQuery 作为接收器多次执行模板

Question

对于 BigQuery 批处理流水线，模板只能执行一次，因为 BigQuery 作业 ID 是在创建模板时设置的。我正在使用 Apache Beam v2.0.0 并且无法多次执行模板。我们可以在头部使用光束来进行此限制吗？如果是的话，我想知道的第一件事是什么是头梁？为了支持多次执行模板，我的 Apache Beam 程序需要进行哪些具体更改？

Maven 依赖：

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-jms</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-examples-java</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-examples-java8</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-common-fn-api</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-build-tools</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-core</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-join-library</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-protobuf</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-sorter</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-amqp</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-jdbc</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-kafka</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-kinesis</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-mongodb</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-mqtt</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-solr</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-core-construction-java</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-core-java</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-direct-java</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-common-runner-api</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>

Answer 1

这是问题BEAM-2058。如果您使用来自Beam github repository 的最新代码，它应该被修复。除了构建新版本的 Beam 并更新您的 pom.xml 以使用它之外，您不需要做任何事情。

或者，等待 Beam 的 2.1.0 版本，该版本正在准备中。