Spring Cloud Data Flow > Stream > Source 在 K8s 中部署但在本地部署时抛出 UNKNOWN_TOPIC_OR_PARTITION答案

【问题标题】：Spring Cloud Data Flow > Stream > Source throws UNKNOWN_TOPIC_OR_PARTITION when deploying in K8s but working in local deploymentSpring Cloud Data Flow > Stream > Source 在 K8s 中部署但在本地部署时抛出 UNKNOWN_TOPIC_OR_PARTITION
【发布时间】：2017-04-07 18:25:36
【问题描述】：

我正在尝试根据http://cloud.spring.io/spring-cloud-dataflow/ 中解释的非常简单的示例运行“Hello, world”Spring Cloud Data Flow 流。我能够创建一个简单的源和接收器，并使用 Kafka 在我的本地 SCDF 服务器上运行它，所以直到这里一切都是正确的。

现在，我正在尝试根据http://docs.spring.io/spring-cloud-dataflow-server-kubernetes/docs/current-SNAPSHOT/reference/htmlsingle/#_getting_started 中列出的说明将其部署到我的私有云中。使用此部署，我可以毫无问题地部署一个简单的“时间|日志”开箱即用流，但我的示例失败了。

具体版本为：

Docker 版本 1.13.1，构建 092cba3
Hyperkube 1.5.5
SCDF 1.2.0.M2
zookeeper 3.4.9-1757313，建于 08/23/2016 06:50 GMT
Kafka 0.10.1.1

源工件日志是：

2017-04-06T11:05:07.429204866Z 2017-04-06 11:05:07,428 INFO main-SendThread(10.0.0.181:2181) o.a.z.ClientCnxn:876 - 已建立到 10.0.0.181/10.0.0.181 的套接字连接:2181，启动会话 2017-04-06T11:05:07.440381666Z 2017-04-06 11:05:07,439 INFO main-SendThread(10.0.0.181:2181) o.a.z.ClientCnxn:1299 - 在服务器 10.0.0.181/10.0.0.181 上完成会话建立：2 , sessionid = 0x15b155ef61e014a, 协商超时 = 10000 2017-04-06T11:05:07.740130495Z 2017-04-06 11:05:07,737 信息主要 o.a.k.c.p.ProducerConfig：180 - ProducerConfig 值： 2017-04-06T11:05:07.740160464Z 确认 = 1 2017-04-06T11:05:07.740163408Z batch.size = 16384 2017-04-06T11:05:07.740165226Z block.on.buffer.full = false 2017-04-06T11:05:07.740166942Z bootstrap.servers = [10.0.0.213:9092] 2017-04-06T11:05:07.740168741Z buffer.memory = 33554432 2017-04-06T11:05:07.740170545Z client.id = 2017-04-06T11:05:07.740172245Z compression.type = none 2017-04-06T11:05:07.740173971Z 连接数.max.idle.ms = 540000 2017-04-06T11:05:07.740175706Z 拦截器.classes = null 2017-04-06T11:05:07.744179899Z reconnect.backoff.ms = 50 2017-04-06T11:05:07.744181600Z request.timeout.ms = 30000 2017-04-06T11:05:07.744183356Z 重试 = 0 2017-04-06T11:05:07.744185083Z retry.backoff.ms = 100 2017-04-06T11:05:07.744186754Z sasl.kerberos.kinit.cmd = /usr/bin/kinit 2017-04-06T11:05:07.744188494Z sasl.kerberos.min.time.before.relogin = 60000 2017-04-06T11:05:07.744190205Z sasl.kerberos.service.name = null 2017-04-06T11:05:07.744191916Z sasl.kerberos.ticket.renew.jitter = 0.05 2017-04-06T11:05:07.744193763Z sasl.kerberos.ticket.renew.window.factor = 0.8 2017-04-06T11:05:07.744195432Z sasl.mechanism = GSSAPI 2017-04-06T11:05:07.744197163Z security.protocol = PLAINTEXT 2017-04-06T11:05:07.744198789Z send.buffer.bytes = 131072 2017-04-06T11:05:07.744200522Z ssl.cipher.suites = null 2017-04-06T11:05:07.744202328Z ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] 2017-04-06T11:05:07.744204161Z ssl.endpoint.identification.algorithm = null 2017-04-06T11:05:07.744205837Z ssl.key.password = null 2017-04-06T11:05:07.744207544Z ssl.keymanager.algorithm = SunX509 2017-04-06T11:05:07.744212464Z ssl.keystore.location = null 2017-04-06T11:05:07.744214272Z ssl.keystore.password = null 2017-04-06T11:05:07.744216025Z ssl.keystore.type = JKS 2017-04-06T11:05:07.744217647Z ssl.protocol = TLS 2017-04-06T11:05:07.744219234Z ssl.provider = null 2017-04-06T11:05:07.744220987Z ssl.secure.random.implementation = null 2017-04-06T11:05:07.744222666Z ssl.trustmanager.algorithm = PKIX 2017-04-06T11:05:07.744224359Z ssl.truststore.location = null 2017-04-06T11:05:07.744226022Z ssl.truststore.password = null 2017-04-06T11:05:07.744228171Z ssl.truststore.type = JKS 2017-04-06T11:05:07.744230006Z timeout.ms = 30000 2017-04-06T11:05:07.744231705Z value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer 2017-04-06T11:05:07.744233544Z 2017-04-06T11:05:07.837193978Z 2017-04-06 11:05:07,834 警告主要 o.a.k.c.p.ProducerConfig:188 - 提供了配置“key.deserializer”，但不是已知配置。 2017-04-06T11:05:07.837221870Z 2017-04-06 11:05:07,835 警告主要 o.a.k.c.p.ProducerConfig:188 - 提供了配置“value.deserializer”，但不是已知配置。 2017-04-06T11:05:07.929207703Z 2017-04-06 11:05:07,926 信息主要 o.a.k.c.u.AppInfoParser:83 - Kafka 版本：0.10.1.1 2017-04-06T11:05:07.929239636Z 2017-04-06 11:05:07,927 信息主要 o.a.k.c.u.AppInfoParser:84 - Kafka commitId : f10ef2720b03b247 2017-04-06T11:05:08.228817026Z 2017-04-06 11:05:08,228 WARN kafka-producer-network-thread | producer-1 o.a.k.c.NetworkClient:600 - 获取相关 id 为 0 的元数据时出错：{output=UNKNOWN_TOPIC_OR_PARTITION} 2017-04-06T11:05:08.436574800Z 2017-04-06 11:05:08,435 WARN kafka-producer-network-thread | producer-1 o.a.k.c.NetworkClient:600 - 获取相关 id 为 1 的元数据时出错：{output=UNKNOWN_TOPIC_OR_PARTITION}

Zookepeer 日志是：

2017-04-06T11:04:38.000953447Z 2017-04-06 11:04:38,000 [myid:] - 信息 [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - 处理的会话终止sessionid: 0x15b155ef61e0148 2017-04-06T11:05:04.939356606Z 2017-04-06 11:05:04,938 [myid:] - 信息 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - 接受来自的套接字连接/10.1.98.5:48180 2017-04-06T11:05:04.940666418Z 2017-04-06 11:05:04,939 [myid:] - 信息 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@928] - 客户端尝试建立新会话在 /10.1.98.5:48180 2017-04-06T11:05:04.943859474Z 2017-04-06 11:05:04,943 [myid:] - 信息 [SyncThread:0:ZooKeeperServer@673] - 已建立会话 0x15b155ef61e0149，客户端 /10.1.98.5 协商超时 10000： 48180 2017-04-06T11:05:07.325929074Z 2017-04-06 11:05:07,325 [myid:] - 信息 [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - sessionid 的已处理会话终止：0x15b155ef61e0149 2017-04-06T11:05:07.342876962Z 2017-04-06 11:05:07,341 [myid:] - 信息 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - 关闭套接字连接客户端 /10.1.98.5:48180 的 sessionid 为 0x15b155ef61e0149 2017-04-06T11:05:07.429909440Z 2017-04-06 11:05:07,429 [myid:] - 信息 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - 接受来自的套接字连接/10.1.98.5:48182 2017-04-06T11:05:07.429933377Z 2017-04-06 11:05:07,429 [myid:] - 信息 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@928] - 客户端尝试建立新会话在 /10.1.98.5:48182 2017-04-06T11:05:07.441158222Z 2017-04-06 11:05:07,439 [myid:] - 信息 [SyncThread:0:ZooKeeperServer@673] - 已建立会话 0x15b155ef61e014a，客户端/10.1.98.5 协商超时 10000： 48182 2017-04-06T11:05:29.695276997Z 2017-04-06 11:05:29,694 [myid:] - 警告 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - 捕获流结束例外 2017-04-06T11:05:29.695325790Z EndOfStreamException：无法从客户端 sessionid 0x15b155ef61e014a 读取其他数据，可能客户端已关闭套接字 2017-04-06T11:05:29.695328912Z 在 org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) 2017-04-06T11:05:29.695331119Z 在 org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203) 2017-04-06T11:05:29.695333009Z 在 java.lang.Thread.run(Thread.java:745) 2017-04-06T11:05:29.696333706Z 2017-04-06 11:05:29,696 [myid:] - 信息 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - 关闭套接字连接客户

在异常发生的那一刻，我在 Kafka 中没有找到任何日志。

源类的代码 sn-p 是

package xxxx;

import java.text.SimpleDateFormat;
import java.util.Date;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.stream.annotation.EnableBinding;
import org.springframework.cloud.stream.messaging.Source;
import org.springframework.context.annotation.Bean;
import org.springframework.integration.annotation.InboundChannelAdapter;
import org.springframework.integration.core.MessageSource;
import org.springframework.messaging.support.GenericMessage;

@SpringBootApplication
@EnableBinding(Source.class)
public class HelloNitesApplication
{
    public static void main(String[] args)
    {
        SpringApplication.run(HelloNitesApplication.class, args);
    }

    @Bean
    @InboundChannelAdapter(value = Source.OUTPUT)
    public MessageSource<String> timerMessageSource()
    {
        return () -> new GenericMessage<>("Hello " + new SimpleDateFormat().format(new Date()));
    }

所以，包含流源的 pod 不断循环崩溃。

【问题讨论】：

标签： apache-kafka kubernetes spring-cloud-stream

【解决方案1】：

问题似乎是我的实现忽略了属性“spring.cloud.stream.bindings.output.destination=XXX”，并且我在执行之前删除了主题“输出”，正如我所期望的那样在属性指定的主题中。

在我重新部署所有内容后，源工作正常，主题创建正确，尽管将消息插入“输出”主题而不是我定义的属性指定的主题。

【讨论】：