来自文本文件 java.lang.NumberFormatException 的 scala RDD：对于输入字符串：“0.5312”

Question

有人对 NumberFormat 异常有一些想法吗？在这种情况下 toInt 不能转换？

movie-ratings.tsv 中的样本数据

1.6339'鳄鱼'邓迪二世 1988
8.4034 13 游戏 sayawng 2006
1.2864 我讨厌你的 10 件事 1999
...

scala> val moviesRDD = spark.sparkContext.textFile("movie-ratings.tsv", 3).map(line => line.split("\\t"))
moviesRDD: org.apache.spark.rdd.RDD[Array[String]] = MapPartitionsRDD[2] at map at <console>:23

scala> val ratesRDD = moviesRDD.map(r => (r(2), r(0).toInt))
ratesRDD: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[24] at map at <console>:25

scala>

scala> val maxRateByYear = ratesRDD.reduceByKey(math.max(_, _))
maxRateByYear: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[25] at reduceByKey at <console>:25

scala>

scala> maxRateByYear.take(1)
21/09/14 17:02:16 ERROR Executor: Exception in task 1.0 in stage 19.0 (TID 38)
java.lang.NumberFormatException: For input string: "0.5312"
        at java.lang.NumberFormatException.forInputString(Unknown Source)
        at java.lang.Integer.parseInt(Unknown Source)
        at java.lang.Integer.parseInt(Unknown Source)
        at scala.collection.immutable.StringLike.toInt(StringLike.scala:304)
        at scala.collection.immutable.StringLike.toInt$(StringLike.scala:304)
        at scala.collection.immutable.StringOps.toInt(StringOps.scala:33)
        at $line41.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.$anonfun$ratesRDD$1(<console>:25)
        at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
        at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:194)
        at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
        at org.apache.spark.scheduler.Task.run(Task.scala:131)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

非常感谢回复

Answer 1

您需要使用toDouble 而不是toInt。

val ratesRDD = moviesRDD.map(r => (r(2), r(0).toDouble))