【发布时间】：2019-05-16 13:28:54
【问题描述】：
training = spark.read.format("libsvm").load("sample_linear_regression_data.txt")
其中sample_linear_regression_data.txt是数据，样本数据是
-9.490009878824548 1:0.4551273600657362 2:0.36644694351969087 3:-0.38256108933468047 4:-0.4458430198517267 5:0.33109790358914726 6:0.8067445293443565 7:-0.2624341731773887
它是 libsvm 格式，但使用上述语法加载数据会给我如下错误
Py4JJavaError: An error occurred while calling o68.load.
: java.lang.UnsupportedOperationException: empty collection
at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$apply$35.apply(RDD.scala:1037)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$apply$35.apply(RDD.scala:1037)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:1037)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.reduce(RDD.scala:1017)
at org.apache.spark.mllib.util.MLUtils$.computeNumFeatures(MLUtils.scala:94)
at org.apache.spark.ml.source.libsvm.LibSVMFileFormat$$anonfun$1.apply$mcI$sp(LibSVMRelation.scala:104)
at org.apache.spark.ml.source.libsvm.LibSVMFileFormat$$anonfun$1.apply(LibSVMRelation.scala:95)
at org.apache.spark.ml.source.libsvm.LibSVMFileFormat$$anonfun$1.apply(LibSVMRelation.scala:95)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.ml.source.libsvm.LibSVMFileFormat.inferSchema(LibSVMRelation.scala:95)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$6.apply(DataSource.scala:180)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$6.apply(DataSource.scala:180)
at scala.Option.orElse(Option.scala:289)
at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:179)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:373)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:844)
【问题讨论】：
-
什么是 Spark 版本？可以试试最新的 2.4.0 吗？它可以在 spark-shell (Scala) 中与 Spark 2.4.0 一起正常工作。