【问题标题】:Can Flink/Delta Connector create multiple databricks tableFlink/Delta Connector可以创建多个databricks表吗
【发布时间】:2022-08-12 18:29:10
【问题描述】:

我正在使用一个新的Flink/Databricks connector,我试图了解单个 Flink 接收器是否可以创建多个增量表?

查看文档,对 path/s3/hdfs 的引用是单个位置

   /**
 * Convenience method for creating a {@link RowDataDeltaSinkBuilder} for {@link DeltaSink} to a
 * Delta table.
 *
 * @param basePath root path of the Delta table
 * @param conf     Hadoop\'s conf object that will be used for creating instances of
 *                 {@link io.delta.standalone.DeltaLog} and will be also passed to the
 *                 {@link ParquetRowDataBuilder} to create {@link ParquetWriterFactory}
 * @param rowType  Flink\'s logical type to indicate the structure of the events in the stream
 * @return builder for the DeltaSink
 */
public static RowDataDeltaSinkBuilder forRowData(
    final Path basePath,
    final Configuration conf,
    final RowType rowType
) {
    return new RowDataDeltaSinkBuilder(
        basePath,
        conf,
        rowType,
        false // mergeSchema
    );

是否可以根据事件流动态更改 basePath?

标签: apache-flink databricks delta-lake databricks-connect


【解决方案1】:

所以一般来说这是不支持的。 Delta/Flink Source 和 Sink 目前都可以读取/写入一个位置(一个表)。

【讨论】:

    猜你喜欢
    • 2019-06-07
    • 2021-04-15
    • 2022-11-05
    • 2021-09-29
    • 2019-05-14
    • 2021-10-30
    • 2020-11-04
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多