使用 zookeeper-shall.sh rmr brokers/topics 删除主题与 Kafka10 上 kafka-topics.sh 上的删除标志之间的区别

Question

根据我在网上和其他堆栈溢出帖子中找到的内容，似乎主要有两种方法可以删除 kafka 上的主题。 第一个是：a）delete.topic.enable = true 并运行./kafka-topics.sh ---delete --topic <topicName> 第二种方式：./zookeeper-shell.sh localhost:2181 rmr brokers/topics

我确实注意到第一种方法将每个主题标记为要删除，并且在几分钟后主题被删除，而第二种方法会立即删除它们。我还注意到重新启动服务器需要几个小时，这正常吗？我在一个代理上拥有超过 1000 个主题（用于测试目的）。

Answer 1

第一种方法将在 zookeper admin/delete_topics/<topic> 中创建一个节点，如果您像以前一样启用了主题删除，kafka 代理 (TopicDeletionManager) 中的一个给定线程，它监控 delete_topics 子节点，将处理这个，这个意味着从 zookeper 中删除，但也从所有 kafka 副本中删除日志，以确保您最终不会处于无效状态。整个过程在这里描述： https://github.com/apache/kafka/blob/0.11.0/core/src/main/scala/kafka/controller/TopicDeletionManager.scala

/**
 * This manages the state machine for topic deletion.
 * 1. TopicCommand issues topic deletion by creating a new admin path /admin/delete_topics/<topic>
 * 2. The controller listens for child changes on /admin/delete_topic and starts topic deletion for the respective topics
 * 3. The controller's ControllerEventThread handles topic deletion. A topic will be ineligible
 *    for deletion in the following scenarios -
  *   3.1 broker hosting one of the replicas for that topic goes down
  *   3.2 partition reassignment for partitions of that topic is in progress
 * 4. Topic deletion is resumed when -
 *    4.1 broker hosting one of the replicas for that topic is started
 *    4.2 partition reassignment for partitions of that topic completes
 * 5. Every replica for a topic being deleted is in either of the 3 states -
 *    5.1 TopicDeletionStarted Replica enters TopicDeletionStarted phase when onPartitionDeletion is invoked.
 *        This happens when the child change watch for /admin/delete_topics fires on the controller. As part of this state
 *        change, the controller sends StopReplicaRequests to all replicas. It registers a callback for the
 *        StopReplicaResponse when deletePartition=true thereby invoking a callback when a response for delete replica
 *        is received from every replica)
 *    5.2 TopicDeletionSuccessful moves replicas from
 *        TopicDeletionStarted->TopicDeletionSuccessful depending on the error codes in StopReplicaResponse
 *    5.3 TopicDeletionFailed moves replicas from
 *        TopicDeletionStarted->TopicDeletionFailed depending on the error codes in StopReplicaResponse.
 *        In general, if a broker dies and if it hosted replicas for topics being deleted, the controller marks the
 *        respective replicas in TopicDeletionFailed state in the onBrokerFailure callback. The reason is that if a
 *        broker fails before the request is sent and after the replica is in TopicDeletionStarted state,
 *        it is possible that the replica will mistakenly remain in TopicDeletionStarted state and topic deletion
 *        will not be retried when the broker comes back up.
 * 6. A topic is marked successfully deleted only if all replicas are in TopicDeletionSuccessful
 *    state. Topic deletion teardown mode deletes all topic state from the controllerContext
 *    as well as from zookeeper. This is the only time the /brokers/topics/<topic> path gets deleted. On the other hand,
 *    if no replica is in TopicDeletionStarted state and at least one replica is in TopicDeletionFailed state, then
 *    it marks the topic for deletion retry.

直接从 zookeeper 中删除只是意味着从 orchestrator 中删除。当然，在请求元数据时，主题不再存在（好吧，也许它们可以从缓存中），但日志文件没有被删除（至少现在不是，我假设代理会检测到日志无效并在某个时间删除它们)，但是您可能对经纪人有些不连贯（如果您处于重新平衡的中间，您可能会破坏很多东西）。这可能意味着一些经纪人会认为它已删除，而另一些经纪人会认为它仍然存在......远非理想。

从现在开始，删除 fom zookeeper（以及来自代理的日志）似乎确实可行，但请注意，它可能会导致冲突、无效状态、随机错误，并且在未来的版本中可能根本不起作用。