Catch 子句不适用于 hive_context.read.json 函数

Question

我有这个 PySpark 脚本：

from pyspark.sql import HiveContext
from pyspark.sql import SQLContext
from pyspark import SparkContext

sc = SparkContext.getOrCreate()
hive_context = HiveContext(sc)

sc.addFile("hdfs:///user/cloudera/2904/src/LogFunction.py")
import LogFunction

try:
    df = hive_context.read.json("/user/cloudera/Projet/pareeam.json")
except IOError:
    LogFunction.WarnLog("Nope")
....
....

这是我的 LogFunction.py：

import logging
from logging.handlers import RotatingFileHandler
from pyspark.sql import HiveContext
from pyspark.sql import SQLContext
from pyspark import SparkContext
sc = SparkContext.getOrCreate()
hive_context = HiveContext(sc)

df = hive_context.read.json("/user/cloudera/2904/param.json")
Path = df[df.column.isin("LogRep")].collect()[0][1]
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s :: %(levelname)s :: %(message)s')

debug_handler = RotatingFileHandler(Path+ '/LogDebug.log', 'a', 1000000, 1)
debug_handler.setLevel(logging.DEBUG)
debug_handler.setFormatter(formatter)
logger.addHandler(debug_handler)
info_handler = RotatingFileHandler(Path+ '/LogInfo.log', 'a', 1000000, 1)
info_handler.setLevel(logging.INFO)
info_handler.setFormatter(formatter)
logger.addHandler(info_handler)
warning_handler = RotatingFileHandler(Path+ '/LogWarning.log', 'a', 1000000, 1)
warning_handler.setLevel(logging.WARNING)
warning_handler.setFormatter(formatter)
logger.addHandler(warning_handler)

error_handler = RotatingFileHandler(Path+ '/LogError.log', 'a', 1000000, 1)
error_handler.setLevel(logging.ERROR)
error_handler.setFormatter(formatter)
logger.addHandler(error_handler)

def WarnLog(a):
    logger.warning(a)

无论如何，我确信我的日志记录正在运行，因为我已经尝试过其他错误（// 除以零异常）。

但这个例子似乎不是这样。在执行脚本时，我得到一个

py4j.protocol.Py4JJavaError: An error occurred while calling o31.json.
: java.io.FileNotFoundException: File hdfs://quickstart.cloudera:8020/user/cloudera/Projet/pareeam.json does not exist.

在终端上，但我的警告文件中没有任何内容。任何帮助为什么它没有捕捉到错误？谢谢

Answer 1

您遇到了错误的异常。您的代码捕获IOError 而DataFrameReader.json 抛出py4j.protocol.Py4JJavaError（内部）和pyspark.sql.utils.AnalysisException

将您的代码更改为

from pyspark.sql.utils import AnalysisException

try:
    df = hive_context.read.json("/does/not/exist")
except AnalysisException:
    LogFunction.WarnLog("Nope")

Answer 2

导入 AnalysisException 后，您应该捕获 IOError

from pyspark.sql.utils import AnalysisException


try:
    df = hive_context.read.json("/does/not/exist")
except IOError:
    LogFunction.WarnLog("Nope")