在 Spark 2 中获取 CastClassException：java.lang.ClassCastException：java.util.ArrayList 无法转换为 org.apache.hadoop.io.Text

Question

在处理具有复杂数据类型列（如 Array 和 Array）的表时，在 Spark 2 中获取 CastClassException

我尝试的操作很简单：计数

df=spark.sql("select * from <tablename>")
df.count    

但在运行 spark 应用程序时出现错误

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 5, sandbox.hortonworks.com, executor 1): java.lang.ClassCastException: java.util.ArrayList cannot be cast to org.apache.hadoop.io.Text
at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)
at org.apache.spark.sql.hive.HiveInspectors$$anonfun$unwrapperFor$23.apply(HiveInspectors.scala:529)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$14$$anonfun$apply$15.apply(TableReader.scala:419)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$14$$anonfun$apply$15.apply(TableReader.scala:419)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:435)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:426)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)

奇怪的是，spark-shell 中数据框的相同操作工作正常

表格有以下复杂的列：

|-- sku_product: array (nullable = true)
|    |-- element: struct (containsNull = true)
|    |    |-- sku_id: string (nullable = true)
|    |    |-- qty: string (nullable = true)
|    |    |-- price: string (nullable = true)
|    |    |-- display_name: string (nullable = true)
|    |    |-- sku_displ_clr_desc: string (nullable = true)
|    |    |-- sku_sz_desc: string (nullable = true)
|    |    |-- parent_product_id: string (nullable = true)
|    |    |-- delivery_mthd: string (nullable = true)
|    |    |-- pick_up_store_id: string (nullable = true)
|    |    |-- delivery: string (nullable = true)
|-- hitid_low: string (nullable = true)
|-- evar7: array (nullable = true)
|    |-- element: string (containsNull = true)
|-- hitid_high: string (nullable = true)
|-- evar60: array (nullable = true)
|    |-- element: string (containsNull = true)

如果需要任何进一步的信息，请告诉我。

Answer 1

我遇到了类似的问题。我正在使用带有镶木地板文件的 spark 2.1。 我发现其中一个镶木地板文件的架构与其他文件不同。所以当我试图阅读所有内容时，我得到了演员错误。 为了解决它，我只是逐个文件检查。