如何使用 KafkaIO 设置 AvroCoder 和使用 Java 设置 Apache Beam

Question

我正在尝试创建一个管道，将数据从 Kafka 主题流式传输到 google 的 Bigquery。主题中的数据在 Avro 中。

我调用了 apply 函数 3 次。一次从 Kafka 读取，一次提取记录，一次写入 Bigquery。这是代码的主要部分：

        pipeline
            .apply("Read from Kafka",
                    KafkaIO
                            .<byte[], GenericRecord>read()
                            .withBootstrapServers(options.getKafkaBrokers().get())
                            .withTopics(Utils.getListFromString(options.getKafkaTopics()))
                            .withKeyDeserializer(
                                    ConfluentSchemaRegistryDeserializerProvider.of(
                                            options.getSchemaRegistryUrl().get(),
                                            options.getSubject().get())
                            )
                            .withValueDeserializer(
                                    ConfluentSchemaRegistryDeserializerProvider.of(
                                            options.getSchemaRegistryUrl().get(),
                                            options.getSubject().get()))
                            .withoutMetadata()
            )

            .apply("Extract GenericRecord",
                    MapElements.into(TypeDescriptor.of(GenericRecord.class)).via(KV::getValue)
            )
            .apply(
                    "Write data to BQ",
                    BigQueryIO
                            .<GenericRecord>write()
                            .optimizedWrites()
                            .useBeamSchema()
                            .useAvroLogicalTypes()
                            .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
                            .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
                            .withSchemaUpdateOptions(ImmutableSet.of(BigQueryIO.Write.SchemaUpdateOption.ALLOW_FIELD_ADDITION))
                            //Temporary location to save files in GCS before loading to BQ
                            .withCustomGcsTempLocation(options.getGcsTempLocation())
                            .withNumFileShards(options.getNumShards().get())
                            .withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())
                            .withMethod(FILE_LOADS)
                            .withTriggeringFrequency(Utils.parseDuration(options.getWindowDuration().get()))
                            .to(new TableReference()
                                    .setProjectId(options.getGcpProjectId().get())
                                    .setDatasetId(options.getGcpDatasetId().get())
                                    .setTableId(options.getGcpTableId().get()))

            );

运行时出现以下错误：

    Exception in thread "main" java.lang.IllegalStateException: Unable to return a default Coder for Extract GenericRecord/Map/ParMultiDo(Anonymous).output [PCollection]. Correct one of the following root causes:  No Coder has been manually specified;  you may do so using .setCoder().
  Inferring a Coder from the CoderRegistry failed: Unable to provide a Coder for org.apache.avro.generic.GenericRecord.
  Building a Coder using a registered CoderProvider failed.

如何设置编码器以正确读取 Avro？

Answer 1

至少有三种方法：

1. 设置内联编码器：

     pipeline.apply("Read from Kafka", ....)  
    .apply("Dropping key", Values.create())
    .setCoder(AvroCoder.of(Schema schemaOfGenericRecord))
    .apply("Write data to BQ", ....);

请注意，该键已被删除，因为它未使用，因此您将不再需要 MapElements。

1. 在管道的 CoderRegistry 实例中注册编码器：

pipeline.getCoderRegistry().registerCoderForClass(GenericRecord.class, AvroCoder.of(Schema genericSchema));

1. 通过以下方式从架构注册表中获取编码器：

ConfluentSchemaRegistryDeserializerProvider.getCoder(CoderRegistry registry)

https://beam.apache.org/releases/javadoc/2.22.0/org/apache/beam/sdk/io/kafka/ConfluentSchemaRegistryDeserializerProvider.html#getCoder-org.apache.beam.sdk.coders.CoderRegistry-