在 IntelliJ 中启用 Spark 日志

Question

我在 intelliJ 上运行我的 spark 作业，并且收到以下错误：-

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/sanchay/brazil-pkg-cache/packages/Log4j-slf4j/Log4j-slf4j-2.x.698.1338/RHEL5_64/DEV.STD.PTHREAD/build/lib/log4j-slf4j-impl-2.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/sanchay/brazil-pkg-cache/packages/Slf4j_log4j/Slf4j_log4j-1.7.2463.17539/RHEL5_64/DEV.STD.PTHREAD/build/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/sanchay/.m2/repository/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.

但是，当我使用 spark-submit 运行相同的 jar/类时，我可以获得所需的所有 INFO 日志：-

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/11/08 00:17:27 INFO SparkContext: Running Spark version 2.2.0
17/11/08 00:17:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/08 00:17:28 INFO SparkContext: Submitted application: JavaSparkPi
17/11/08 00:17:28 INFO SecurityManager: Changing view acls to: sanchay
17/11/08 00:17:28 INFO SecurityManager: Changing modify acls to: sanchay
17/11/08 00:17:28 INFO SecurityManager: Changing view acls groups to: 
17/11/08 00:17:28 INFO SecurityManager: Changing modify acls groups to: 
17/11/08 00:17:28 INFO SecurityManager: SecurityManager: authentication dis

如何切换到 IntelliJ 中的默认日志文件，以便通过 INFO 日志获取有关 Spark Ui 等的信息？

Answer 1

• 假设您使用 maven 管理代码。构建依赖：树并检查使用的日志库版本。 https://www.slf4j.org/legacy.html 将提供非常好的图表，其中包含库应该是什么的信息。应排除所有其他人

• 如何构建要提交给 spark 的 jar？我假设您没有包含所有依赖项，因此您没有额外的日志库。