停止 Spark Streaming 中的默认输出答案

【问题标题】：Stopping the Default Output in Spark Streaming停止 Spark Streaming 中的默认输出
【发布时间】：2018-07-17 06:27:17
【问题描述】：

有没有办法禁用Spark streaming 中显示的默认输出。下面给出的是运行火花流应用程序时打印的默认输出。我希望只打印我添加的日志消息（在驱动程序中）

spark-submit --class a11_TCP_IP.a11_Basic.a11_HelloTcp target/scala-2.10/sparkstreamingexamples_2.10-1.0.jar

16/04/27 13:20:15 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data
16/04/27 13:20:15 ERROR ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Socket data stream had no more data
16/04/27 13:20:17 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data
16/04/27 13:20:17 ERROR ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Socket data stream had no more data
-------------------------------------------
Time: 1461777618000 ms
-------------------------------------------

16/04/27 13:20:19 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data
16/04/27 13:20:19 ERROR ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Socket data stream had no more data
-------------------------------------------
Time: 1461777621000 ms
-------------------------------------------

16/04/27 13:20:21 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data

【问题讨论】：

标签： apache-spark spark-streaming

【解决方案1】：

您的 Spark 文件夹应该有一个 conf/ 目录。在其中，有一个log4j.properties 文件。如果没有，请将conf/log4j.properties.template 复制到conf/log4j.properties。

现在在编辑器中打开conf/log4j.properties，并替换这一行：

log4j.rootCategory=INFO, console

这两个：

log4j.rootCategory=INFO, null
log4j.appender.null=org.apache.log4j.varia.NullAppender

【讨论】：

谢谢。这确实会处理错误和警告。但我仍然得到这个输出------------------------------------------- Time: 1462045608000 ms -------------------------------------------
好的。我终于确定了。 DStream的输出操作； Dstream.print() 默认提供该输出（即---- Time: .... ）。我使用了另一个输出操作DStream.foreachRDD(..) 帮助解决了这个问题

【解决方案2】：

我正在使用 IntelliJ 进行 Spark Streaming，但看不到 conf 文件夹。所以使用代码来调整日志记录。

Logger.getLogger("org").setLevel(Level.ERROR) //导入需要的库

【讨论】：