Scala 单元测试中的 ExceptionInInitializerError (Scalacheck, Scalatest)

Question

我已经编写了引用 DataframeGenerator example 的单元测试，它允许您动态生成模拟数据帧

成功执行以下命令后

sbt clean
sbt update
sbt compile

运行以下任一命令时，我得到输出中显示的错误

sbt assembly
sbt test -- -oF

输出

...
[info] SearchClicksProcessorTest:
17/11/24 14:19:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/24 14:19:07 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect.
17/11/24 14:19:18 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
17/11/24 14:19:18 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
17/11/24 14:19:19 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
[info] - testExplodeMap *** FAILED ***
[info]   ExceptionInInitializerError was thrown during property evaluation.
[info]     Message: "None"
[info]     Occurred when passed generated values (
[info]   
[info]     )
[info] - testFilterByClicks *** FAILED ***
[info]   NoClassDefFoundError was thrown during property evaluation.
[info]     Message: Could not initialize class org.apache.spark.rdd.RDDOperationScope$
[info]     Occurred when passed generated values (
[info]   
[info]     )
[info] - testGetClicksData *** FAILED ***
[info]   NoClassDefFoundError was thrown during property evaluation.
[info]     Message: Could not initialize class org.apache.spark.rdd.RDDOperationScope$
[info]     Occurred when passed generated values (
[info]   
[info]     )
...
[info] *** 3 TESTS FAILED ***
[error] Failed: Total 6, Failed 3, Errors 0, Passed 3
[error] Failed tests:
[error]         com.company.spark.ml.pipelines.search.SearchClicksProcessorTest
[error] (root/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 73 s, completed 24 Nov, 2017 2:19:28 PM

我尝试失败的事情

• 使用 F 标志运行 sbt 测试以显示完整的堆栈跟踪（没有堆栈跟踪输出，如上所示）
• 在 IntelliJ Idea 中重新构建项目

我的问题是

• 此错误的可能原因是什么？
• 如何使 SBT 中的堆栈跟踪输出能够对其进行调试？

---

EDIT-1 我的单元测试类包含如下几种方法

class SearchClicksProcessorTest extends FunSuite with Checkers {
  import spark.implicits._

  test("testGetClicksData") {
    val schemaIn = StructType(List(
      StructField("rank", IntegerType),
      StructField("city_id", IntegerType),
      StructField("target", IntegerType)
    ))
    val schemaOut = StructType(List(
      StructField("clicked_res_rank", IntegerType),
      StructField("city_id", IntegerType),
    ))
    val dataFrameGen = DataframeGenerator.arbitraryDataFrame(spark.sqlContext, schemaIn)

    val property = Prop.forAll(dataFrameGen.arbitrary) { dfIn: DataFrame =>
      dfIn.cache()
      val dfOut: DataFrame = dfIn.transform(SearchClicksProcessor.getClicksData)

      dfIn.schema === schemaIn &&
        dfOut.schema === schemaOut &&
        dfIn.filter($"target" === 1).count() === dfOut.count()
    }
    check(property)
  }
}

而build.sbt 看起来像这样

// core settings
organization := "com.company"
scalaVersion := "2.11.11"

name := "repo-name"
version := "0.0.1"

// cache options
offline := false
updateOptions := updateOptions.value.withCachedResolution(true)

// aggregate options
aggregate in assembly := false
aggregate in update := false

// fork options
fork in Test := true

//common libraryDependencies
libraryDependencies ++= Seq(
  scalaTest,
  typesafeConfig,
  ...
  scalajHttp
)

libraryDependencies ++= allAwsDependencies
libraryDependencies ++= SparkDependencies.allSparkDependencies

assemblyMergeStrategy in assembly := {
  case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
  ...
  case _ => MergeStrategy.first
}

lazy val module-1 = project in file("directory-1")

lazy val module-2 = (project in file("directory-2")).
  dependsOn(module-1).
  aggregate(module-1)

lazy val root = (project in file(".")).
  dependsOn(module-2).
  aggregate(module-2)

Answer 1

附：在阅读此答案之前请阅读comments on original question

---

• 即使是overriding SBT's transitive dependency 而不是faster-xml.jackson 的流行解决方案也不适合我；因为需要进行更多更改（ExceptionInInitializerError 已消失，但出现了其他一些错误）

• 最后（除了上面提到的修复）我最终以不同的方式创建了DataFrames（与此处使用的StructType 相反）。我将它们创建为

  spark.sparkContext.parallelize(Seq(MyType)).toDF()

  其中MyType 是case class，根据DataFrame 的架构

• 在实施此解决方案时，我遇到了一个小问题，虽然从 case class 生成的 schema 的 datatypes 是正确的，但字段的 nullability 是经常不匹配；已找到此问题的修复程序 here

---

在这里我公然承认我不确定什么是正确的解决方法：faster-xml.jackson 依赖或创建DataFrame 的替代方式，所以请随时填补失误理解/调查问题

Answer 2

我遇到过类似的问题案例，经过调查发现，在val 之前添加lazy 解决了我的问题。我的估计是，使用 Scalatest 运行 Scala 程序会调用一些不同的初始化序列。正常的 scala 执行以源代码行号自上而下的顺序初始化 vals - 嵌套的 object {...} 块以相同的方式初始化 - 使用与 Scalatest 相同的编码，执行在嵌套的 @ 中初始化 valss 987654326@ 块之前 vals 行号明智上方 object { ... }。

我知道这绝对是模糊的，但是用 vals 和 lazy 前缀来推迟初始化可以解决这里的测试问题。

这里的关键是它不会发生在正常执行中，只会发生在测试执行中，而在我的情况下，它只会在以这种形式使用带有水龙头的 lambdas 时发生：

...
.tap(x =>
        hook_feld_erweiterungen_hook(
          abc = theProblematicVal
        )
      )
...