无法找到 PySpark 标准输出日志

Question

我正在开发一个 PySpark 应用程序，并以纱线集群模式部署它。我已将标准输出作为日志流处理程序。我可以在 YARN UI 中看到日志。但是，我在 /var/log/sparkapp/yarn 下找不到 stdout 日志。我在那里只看到 stderr 日志。这可能是什么原因？

这是我在应用程序中的日志记录部分

import logging
import sys

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
lsh = logging.StreamHandler(sys.stdout)
lsh.setLevel(logging.INFO)
lformat = logging.Formatter(fmt='%(asctime)s.%(msecs)03d %(levelname)s :%(name)s - %(message)s', datefmt='%m/%d/%Y %I:%M:%S')
lsh.setFormatter(lformat)
logger.addHandler(lsh)

log4j.properties

log4jspark.root.logger=INFO,console
log4jspark.log.dir=.
log4jspark.log.file=spark.log
log4jspark.log.maxfilesize=1024MB
log4jspark.log.maxbackupindex=10

# Define the root logger to the system property "spark.root.logger".
log4j.rootLogger=${log4jspark.root.logger}, EventCounter

# Set everything to be logged to the console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
log4j.appender.console.Threshold=INFO

# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO

Answer 1

试试这个来为你的 spark 作业获取记录器：

log4jLogger = sc._jvm.org.apache.log4j
logger = log4jLogger.LogManager.getLogger(__name__)

您可以修改log4j.properties来更改target文件：

log4j.appender.console.target=System.out