Apache Beam Java（SDK 版本 2.43.0 - 2.44.0）使用 Storage Write API 批量加载到 BigQuery 失败

Question

在我从版本 2.42 升级到 2.43 及更高版本之前，我有一个批处理的 Apache Beam 管道一直在成功运行。管道使用 Storage Write API 写入 Biguery，现在失败并显示错误："Error message from worker: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ExecutionError:jva.lang.NoSuchMethodError: 'com.google.cloud.biguery.storage.v1.StreamWriter$Builder com.google.cloud.biguery.storage.v1.StreamWriter$Builder.setEnableConnectionPool(bool)'"。

当我切换到 FileLoads 方法而不是 StorageWriteAPI 时，它再次起作用。但是想知道 StorageWriteAPI 现在有什么问题。尝试查看发行说明但找不到任何有用的信息 (https://github.com/apache/beam/releases)。我可以在这里得到一些帮助吗？

Answer 1

我认为您在 Maven pom 或 Gradle 构建文件中与 Guava 存在依赖冲突。我与您分享我在我的 pom 文件中用于 Beam 2.43.0 或 2.44.0 版本的 Guava 版本：

<guava.version>31.0.1-jre</guava.version>

如果您使用以下依赖项，则必须排除旧版本的 Guava ：

<dependency>
    <groupId>com.google.api-client</groupId>
    <artifactId>google-api-client</artifactId>
    <version>${google-api-client.version}</version>
    <exclusions>
        <!-- Exclude an old version of guava that is being pulled
             in by a transitive dependency of google-api-client -->
        <exclusion>
            <groupId>com.google.guava</groupId>
            <artifactId>guava-jdk5</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>com.google.apis</groupId>
    <artifactId>google-api-services-bigquery</artifactId>
    <version>${bigquery.version}</version>
    <exclusions>
        <!-- Exclude an old version of guava that is being pulled
             in by a transitive dependency of google-api-client -->
        <exclusion>
            <groupId>com.google.guava</groupId>
            <artifactId>guava-jdk5</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>com.google.http-client</groupId>
    <artifactId>google-http-client</artifactId>
    <exclusions>
        <!-- Exclude an old version of guava that is being pulled
             in by a transitive dependency of google-api-client -->
        <exclusion>
            <groupId>com.google.guava</groupId>
            <artifactId>guava-jdk5</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>com.google.apis</groupId>
    <artifactId>google-api-services-pubsub</artifactId>
    <version>${pubsub.version}</version>
    <exclusions>
        <!-- Exclude an old version of guava that is being pulled
             in by a transitive dependency of google-api-client -->
        <exclusion>
            <groupId>com.google.guava</groupId>
            <artifactId>guava-jdk5</artifactId>
        </exclusion>
    </exclusions>
</dependency>

我希望它能有所帮助。