【问题标题】:io.confluent.ksql.exception.KafkaTopicExistsException: when launching ksql-server-start ksql-server.propertiesio.confluent.ksql.exception.KafkaTopicExistsException:启动 ksql-server-start ksql-server.properties 时
【发布时间】:2020-05-14 00:50:56
【问题描述】:

我使用 ksql 已经有一段时间了。 Kafka 集群(如果有 3 个节点)。我也一直在使用 udf 并且一切看起来都很好,直到我停止服务器并重新启动它们。 在服务器启动时,我在日志中看到以下内容:

[2019-04-03 11:29:54,381] ERROR Exception encountered running command: A Kafka topic with the name 'czxcorp-structured-data-enriched' already exists, with different partition/replica configuration than required. KSQL expects 4 partitions (topic has 9), and 1 replication factor (topic has 1).. Retrying in 5000 ms (io.confluent.ksql.util.RetryUtil:80)
[2019-04-03 11:29:54,381] ERROR Stack trace: io.confluent.ksql.exception.KafkaTopicExistsException: A Kafka topic with the name 'czxcorp-structured-data-enriched' already exists, with different partition/replica configuration than required. KSQL expects 4 partitions (topic has 9), and 1 replication factor (topic has 1).
        at io.confluent.ksql.services.TopicValidationUtil.validateTopicProperties(TopicValidationUtil.java:51)
        at io.confluent.ksql.services.TopicValidationUtil.validateTopicProperties(TopicValidationUtil.java:35)
        at io.confluent.ksql.services.KafkaTopicClientImpl.validateTopicProperties(KafkaTopicClientImpl.java:292)
        at io.confluent.ksql.services.KafkaTopicClientImpl.createTopic(KafkaTopicClientImpl.java:76)
        at io.confluent.ksql.planner.plan.KsqlStructuredDataOutputNode.createSinkTopic(KsqlStructuredDataOutputNode.java:244)
        at io.confluent.ksql.planner.plan.KsqlStructuredDataOutputNode.buildStream(KsqlStructuredDataOutputNode.java:146)
        at io.confluent.ksql.physical.PhysicalPlanBuilder.buildPhysicalPlan(PhysicalPlanBuilder.java:106)
        at io.confluent.ksql.QueryEngine.buildPhysicalPlan(QueryEngine.java:113)
        at io.confluent.ksql.KsqlEngine$EngineExecutor.execute(KsqlEngine.java:625)
        at io.confluent.ksql.KsqlEngine$EngineExecutor.access$800(KsqlEngine.java:577)
        at io.confluent.ksql.KsqlEngine.execute(KsqlEngine.java:247)
        at io.confluent.ksql.rest.server.computation.StatementExecutor.startQuery(StatementExecutor.java:277)
        at io.confluent.ksql.rest.server.computation.StatementExecutor.executeStatement(StatementExecutor.java:191)
        at io.confluent.ksql.rest.server.computation.StatementExecutor.handleStatementWithTerminatedQueries(StatementExecutor.java:167)
        at io.confluent.ksql.rest.server.computation.StatementExecutor.handleRestore(StatementExecutor.java:101)
        at io.confluent.ksql.rest.server.computation.CommandRunner.lambda$null$0(CommandRunner.java:139)
        at io.confluent.ksql.util.RetryUtil.retryWithBackoff(RetryUtil.java:63)
        at io.confluent.ksql.util.RetryUtil.retryWithBackoff(RetryUtil.java:36)
        at io.confluent.ksql.rest.server.computation.CommandRunner.lambda$processPriorCommands$1(CommandRunner.java:135)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at io.confluent.ksql.rest.server.computation.CommandRunner.processPriorCommands(CommandRunner.java:134)
        at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:414)
        at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:80)
        at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:42)
 (io.confluent.ksql.util.RetryUtil:84)

虽然我已经停止/终止了所有查询,但日志会打印我从一开始为测试直到数据执行的所有命令,包括create, select, drop。我已经从 /ext 文件夹中取出 .jar(UDF) 并启动了服务器,尽管日志打印 udf 功能(我正在使用)不可用。

这是我的 ksql-server.properties:

bootstrap.servers=hostname:9092
service.id=cyan_ksql
commit.interval.ms=5000
cache.max.bytes.buffering=20000000
num.stream.threads=10
fail.on.deserialization.error=false
listeners=http://localhost:8088
ksql.extension.dir=/opt/ksql-master/ext/

对错误发疯了。我正在删除该主题并以某种方式重新创建了该主题。有人请帮忙。

【问题讨论】:

    标签: apache-kafka ksqldb kafka-topic


    【解决方案1】:

    检查错误:

    A Kafka topic with the name 'czxcorp-structured-data-enriched' already exists, with different partition/replica configuration than required. 
    KSQL expects 4 partitions (topic has 9), and 1 replication factor (topic has 1)
    

    如果您已删除该主题,则要么

    • 实际上并没有被删除
    • 它被删除了,其他东西用九个分区重新创建了它,你的错误 KSQL 查询没有指定覆盖 (WITH (PARTITIONS=9) 到默认的四个
    • 另一个 KSQL 命令在出错的命令之前创建它,并且您的错误 KSQL 查询没有指定覆盖 (WITH (PARTITIONS=9) 到默认的四个

    如果您想摆脱状态并从头开始,只需更改您的 ksql.service.id,这将导致 KSQL 使用新的命令主题(这是在您重新启动进程时重播的内容)

    【讨论】:

    • 美丽。它适用于 ksql.service.id 的更改。谢谢罗宾。我已经更新了这个问题,请您在 ksql-server.properties 中提出更正建议吗?谢谢
    • 您可以在属性文件中更改service.id - 这就是您的意思吗?不知道你在问什么
    • 好吧,我更改了 service.id 并且 ksql 服务器从头开始。完全没有问题。我的问题是除了原始问题中曾经提到的为了更好的性能需要的任何更改/附加参数吗?
    • 我建议您提出一个新问题,并详细说明您遇到并希望解决的性能问题。谢谢。
    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 2020-11-21
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多