将向量转换为数据框时出错[重复]答案

【问题标题】：Error while converting Vector to dataframe [duplicate]将向量转换为数据框时出错[重复]
【发布时间】：2018-04-06 23:30:13
【问题描述】：

将 Vector 转换为数据框时出错

第一部分中提到的代码运行良好，但是将矢量数据转换为数据框是一种不直观的方式。

我想用我知道的解决这个问题，即第二部分中提到的代码。你能帮忙吗

val data = Seq(
      Vectors.sparse(4, Seq((0, 1.0), (3, -2.0))),
      Vectors.dense(4.0, 5.0, 0.0, 3.0),
      Vectors.dense(6.0, 7.0, 0.0, 8.0),
      Vectors.sparse(4, Seq((0, 9.0), (3, 1.0)))
    )   

val tupleList = data.map(Tuple1.apply)
val df = rdd.toDF("features")

我们不能像下面这样简单吗

    val rdd = sc.parallelize(data).map(a => Row(a))
rdd.take(1)

val fields = "features".split(" ").map(fields => StructField(fields,DoubleType, nullable =true))
val df = spark.createDataFrame(rdd, StructType(fields))
df.count()

但我收到如下错误

df: org.apache.spark.sql.DataFrame = [features: double]
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 357.0 failed 4 times, most recent failure: Lost task 1.3 in stage 357.0 (TID 1243, datacouch, executor 3): java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: org.apache.spark.ml.linalg.DenseVector is not a valid external type for schema of double
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, features), DoubleType) AS features#6583
    at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:290)
    at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:586)
    at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:586)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithoutKey$(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)

【问题讨论】：

@user8371915 请先阅读我的问题

标签： apache-spark machine-learning spark-dataframe apache-spark-mllib

【解决方案1】：

正如VectorUDT usage 中明确解释的那样，在例外情况下，Vector 的正确DataType 是org.apache.spark.ml.linalg.SQLDataTypes.VectorType：

spark.createDataFrame(
  rdd, 
  StructType(Seq(
    StructField("features", org.apache.spark.ml.linalg.SQLDataTypes.VectorType)
  ))
)

【讨论】：