在 Windows 上使用“storm jar”命令远程部署 Storm 拓扑答案

【问题标题】：Deploy Storm topology remotely using "storm jar" command on Windows在 Windows 上使用“storm jar”命令远程部署 Storm 拓扑
【发布时间】：2015-10-06 17:40:11
【问题描述】：

我正在尝试在 Windows 上使用storm jar 命令将风暴拓扑部署到远程机器

storm jar jarName.jar mainClass

这是我的storm.yaml 文件中的内容：

storm.zookeeper.servers:
     - "HostName"

nimbus.host: "HostName"
nimbus.thrift.port: 6627

但每当我尝试部署时，它总是会尝试部署到我的本地机器上。

编辑

我尝试了以下配置，但它不起作用：

conf.put(Config.NIMBUS_HOST, "HostName");
conf.put(Config.NIMBUS_THRIFT_PORT,6627);
conf.put(Config.STORM_ZOOKEEPER_SERVERS,Arrays.asList(new String[]{"HostName"}));
conf.put(Config.STORM_ZOOKEEPER_PORT,2181);

我还在命令行上尝试了 -c 选项：

storm -c nimbus.host=HostName jar jarName.jar mainClass

但它不起作用；我明白了：

Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

我尝试将 -c 部分放在不同的地方，但似乎没有任何效果。

我也明白了

The field Config.NIMBUS_HOST is deprecated

当我设置时：

conf.put(Config.NIMBUS_HOST, "HostName");

我使用的是 Storm 版本 0.10.0-beta

EDIT2

在我使用后从 IDE 提交工作正常

System.setProperty("storm.jar", "path to jar");

但是，它在命令行中不起作用。

编辑3

感谢马特的帮助。我能够使用 eclipse 执行的 java 命令部署拓扑。但是，我仍然无法使用storm jar 命令进行部署。它抱怨拓扑已经存在，即使我在 Storm UI 上看不到任何东西。此外，当我使用 Eclipse 或命令行中的 Java 命令进行部署时，拓扑会成功部署。我应该如何比较执行 java 命令和storm jar 命令在做什么之间的区别？我试图在 cmd 文件的这一部分中回显变量

 if %storm-command% == jar (
    set STORM_OPTS=%STORM_CLIENT_OPTS% %STORM_OPTS% -Dstorm.jar=%2
    echo %STORM_OPTS%
    set CLASSPATH=%CLASSPATH%;%2

    echo %CLASSPATH%

    set CLASS=%3
    set args=%4
    goto start
    :start
    shift
    if [%4] == [] goto done
    set args=%args% %4

    echo args: %args%

    goto start

    :done
    echo %args%
    set storm-command-arguments=%args%

  )

但我没有看到任何异常。另外，我在堆栈跟踪中看到以下异常

    Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])

通知

 topologies:[]

然后它抱怨拓扑名称存在

Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
        at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
        at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)

        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)

        at mainClass.submitTopology(mainClass.java:127)
        at mainClass.main(mainClass.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
        at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
        at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:613)
        at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:549)
        at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)

        at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16546)
        at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16531)
        at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
        at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)

        at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
        at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
        at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
        ... 5 more

最后，这是整个堆栈跟踪。

 X:\abc\xyz\target>ERROR StatusLogger Invalid URL C:/asd/ewt/apache-storm 0.10.0-beta/log4j2/cluster.xml java.net.MalformedURLException: unknown protocol: c
            at java.net.URL.<init>(URL.java:593)
            at java.net.URL.<init>(URL.java:483)
            at java.net.URL.<init>(URL.java:432)
            at java.net.URI.toURL(URI.java:1089)
            at org.apache.logging.log4j.core.config.ConfigurationFactory.getInputFromUri(ConfigurationFactory.java:296)
            at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:395)
            at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:254)
            at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:419)
            at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:138)
            at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:147)
            at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
            at org.apache.logging.log4j.LogManager.getContext(LogManager.java:175)
            at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:102)
            at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
            at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
            at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
            at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:277)
            at org.apache.log4j.Category.<init>(Category.java:56)
            at org.apache.log4j.Logger.<init>(Logger.java:35)
            at org.apache.log4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:59)
            at org.apache.log4j.Logger.getLogger(Logger.java:39)
            at mainClass.<clinit>(mainClass.java:32)
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field nimbus_uptime_secs' is unset!Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
                at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
                at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
                at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)

                at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)

                at mainClass.submitTopology(mainClass.java:127)
                at mainClass.main(mainClass.java:57)
        Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
                at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
                at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
        read(ClusterSummary.java:613)
                at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
        read(ClusterSummary.java:549)
                at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)

                at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
        resultStandardScheme.read(Nimbus.java:16546)
                at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
        resultStandardScheme.read(Nimbus.java:16531)
                at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
                at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)

                at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
                at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
                at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
                ... 5 more

顺便说一句，堆栈跟踪顶部的奇怪日志记录异常/错误是什么？这可能与storm jar 命令的奇怪行为有关吗？

如果我这样做了

java -cp jarName.jar;C:\abc\asdasd\apache-storm-0.10.0-beta\lib\* mainClass

我明白了

513  [main] INFO  b.s.u.Utils - Using defaults.yaml from resources
582  [main] INFO  b.s.u.Utils - Using defaults.yaml from resources
590  [main] INFO  b.s.StormSubmitter - Generated ZooKeeper secret payload for MD
5-digest: -6100635753051151148:-8346710213172614614
591  [main] INFO  b.s.s.a.AuthUtils - Got AutoCreds []
599  [main] INFO  b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeM
s [2000] the maxSleepTimeMs [60000] the maxRetries [5]
626  [main] INFO  b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeM
s [2000] the maxSleepTimeMs [60000] the maxRetries [5]
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protoc
ol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName, uptime_secs:83775, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
        at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
        at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)

        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)

        at com.rbccm.th2015.storm.SampleTopologyX.submitTopology(SampleTopologyX.java:127)
        at com.rbccm.th2015.storm.SampleTopologyX.main(SampleTopologyX.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName,uptime_secs:83775, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
        at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
        at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:613)
        at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:549)
        at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)

        at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16546)
        at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16531)
        at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
        at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)

        at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
        at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
        at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
        ... 5 more

所以它抱怨同样的事情，但通过不同的途径。这可能与我的 YAML 有关吗？ nimbus_uptime_secs 的业务是什么？

最终编辑

好的，该休息一下了。至少目前看来，sotrm jar 命令似乎不适用于 Windows。我可以清楚地看到storm.py 尝试在Windows 中执行的命令和它尝试在Linux 中执行的命令的区别。基本上，路径在 Windows 中被搞砸了，这一定是导致所有这些奇怪异常的原因。

感谢马特提供的所有帮助和支持。我真的学到了很多。

【问题讨论】：

在我使用 System.setProperty("storm.jar", "path to jar");不能从命令行工作
它抱怨拓扑已经存在。我在风暴用户界面或风暴列表命令上看不到任何内容。在我的本地或远程机器上
所以我可以启用本地日志记录，但我应该在哪里寻找它试图上传到的集群？
抱歉，无法使用工作聊天。拒绝访问该位置
另外，我可以看到原因：org.apache.thrift7.protocol.TProtocolException：必填字段“nimbus_uptime_secs”未设置！

标签： java apache-storm

【解决方案1】：

通过storm jar 运行拓扑仅适用于 Linux（但不适用于 Windows）。 “storm”命令是一个 Python 脚本，假定在 Linux 上运行（Storm 官方不支持 Window）。

但是，“storm”命令仅启动具有适当设置的常规java 命令。如果拓扑是通过 IDE 提交的，IDE 会做同样的事情。如果你使用 Eclipse，你可以按照Eclipse: export running configuration中的描述提取使用的配置

注意，需要设置JVM参数-Dstorm.jar=path\to\myJar.jar；否则，StormSubmitter 将找不到应该提交给 Storm 集群的 jar。

提取java 命令后，它应该允许您在Windows 上通过命令行提交拓扑。如果您的 IDE 不支持提取命令，您仍然可以手动将其放在一起（即使这是一个繁琐的解决方案）。

会是这样的：

java -Dstorm.jar=yourJarFile.jar
     -classpath yourJarFile.jar:
                /home/mjsax/.m2/repository/org/apache/storm/storm-core/0.11.0-SNAPSHOT/storm-core-0.11.0-SNAPSHOT.jar:
                /home/mjsax/.m2/repository/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar:
                /home/mjsax/.m2/repository/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar:
                /home/mjsax/.m2/repository/org/ow2/asm/asm/4.0/asm-4.0.jar:
                /home/mjsax/.m2/repository/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar:
                /home/mjsax/.m2/repository/org/objenesis/objenesis/1.2/objenesis-1.2.jar:
                /home/mjsax/.m2/repository/org/clojure/clojure/1.6.0/clojure-1.6.0.jar:
                /home/mjsax/.m2/repository/clj-time/clj-time/0.8.0/clj-time-0.8.0.jar:
                /home/mjsax/.m2/repository/joda-time/joda-time/2.3/joda-time-2.3.jar:
                /home/mjsax/.m2/repository/compojure/compojure/1.1.3/compojure-1.1.3.jar:
                /home/mjsax/.m2/repository/org/clojure/core.incubator/0.1.0/core.incubator-0.1.0.jar:
                /home/mjsax/.m2/repository/org/clojure/tools.macro/0.1.0/tools.macro-0.1.0.jar:
                /home/mjsax/.m2/repository/clout/clout/1.0.1/clout-1.0.1.jar:
                /home/mjsax/.m2/repository/ring/ring-core/1.1.5/ring-core-1.1.5.jar:
                /home/mjsax/.m2/repository/commons-codec/commons-codec/1.6/commons-codec-1.6.jar:
                /home/mjsax/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:
                /home/mjsax/.m2/repository/commons-fileupload/commons-fileupload/1.2.1/commons-fileupload-1.2.1.jar:
                /home/mjsax/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:
                /home/mjsax/.m2/repository/hiccup/hiccup/0.3.6/hiccup-0.3.6.jar:
                /home/mjsax/.m2/repository/ring/ring-devel/1.3.0/ring-devel-1.3.0.jar:
                /home/mjsax/.m2/repository/clj-stacktrace/clj-stacktrace/0.2.7/clj-stacktrace-0.2.7.jar:
                /home/mjsax/.m2/repository/ns-tracker/ns-tracker/0.2.2/ns-tracker-0.2.2.jar:
                /home/mjsax/.m2/repository/org/clojure/tools.namespace/0.2.4/tools.namespace-0.2.4.jar:
                /home/mjsax/.m2/repository/org/clojure/java.classpath/0.2.2/java.classpath-0.2.2.jar:
                /home/mjsax/.m2/repository/ring/ring-jetty-adapter/1.3.0/ring-jetty-adapter-1.3.0.jar:
                /home/mjsax/.m2/repository/ring/ring-servlet/1.3.0/ring-servlet-1.3.0.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-server/7.6.13.v20130916/jetty-server-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/orbit/javax.servlet/2.5.0.v201103041518/javax.servlet-2.5.0.v201103041518.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-continuation/7.6.13.v20130916/jetty-continuation-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-http/7.6.13.v20130916/jetty-http-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-io/7.6.13.v20130916/jetty-io-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/org/eclipse/jetty/jetty-util/7.6.13.v20130916/jetty-util-7.6.13.v20130916.jar:
                /home/mjsax/.m2/repository/ring/ring-json/0.3.1/ring-json-0.3.1.jar:
                /home/mjsax/.m2/repository/cheshire/cheshire/5.3.1/cheshire-5.3.1.jar:
                /home/mjsax/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.3.1/jackson-core-2.3.1.jar:
                /home/mjsax/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.3.1/jackson-dataformat-smile-2.3.1.jar:
                /home/mjsax/.m2/repository/tigris/tigris/0.1.1/tigris-0.1.1.jar:
                /home/mjsax/.m2/repository/org/clojure/tools.logging/0.2.3/tools.logging-0.2.3.jar:
                /home/mjsax/.m2/repository/com/googlecode/disruptor/disruptor/2.10.4/disruptor-2.10.4.jar:
                /home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-api/2.1/log4j-api-2.1.jar:
                /home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-core/2.1/log4j-core-2.1.jar:
                /home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.1/log4j-slf4j-impl-2.1.jar:
                /home/mjsax/.m2/repository/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar:
                /home/mjsax/.m2/repository/org/slf4j/log4j-over-slf4j/1.6.6/log4j-over-slf4j-1.6.6.jar:
                /home/mjsax/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:
                /home/mjsax/.m2/repository/args4j/args4j/2.0.16/args4j-2.0.16.jar
    package.to.your.MainClass

【讨论】：

java -Dstorm.jar="X:\abc\xyz\ppp\ooo\zzz\target\jarName.jar" -cp X:\abc\xyz\ppp\ooo\zzz\target \classes\:X:\abc\xyz\apache-storm-0.10.0-beta\lib* z.x.c.mainClass
为什么不简单地使用“java jar”而不是“storm jar”？？
阅读我的答案....您需要发出相同的命令 Eclipse 问题...它比您的命令更复杂。类路径必须包含更多的库...
您是否使用 Eclipse 作为 IDE？您是否尝试运行 IDE 使用的 java 命令？这应该在 Windows 命令行上工作......（我现在只是好奇；））
是的，eclipse 命令在转换为 jar 文件的 java 执行时起作用。但是storm jar 命令或storm.py jar 命令不起作用。他们都抱怨拓扑名称存在