Spark：java.lang.OutOfMemoryError：超出 GC 开销限制

Question

我有以下代码来转换我从输入文件中读取的数据并创建一个pairedrdd，然后将其转换为一个映射以供将来查找。然后我映射这个广播变量。这是几GB的地图。有没有办法以更有效的方式执行collectAsMap() 或将其替换为其他调用？

val result_paired_rdd = prods_user_flattened.collectAsMap() 

sc.broadcast(result_paired_rdd)

我收到以下错误。我还尝试了以下参数：--executor-memory 7G 和 spark-submit 命令。

15/08/31 08:29:51 INFO BlockManagerInfo: Removed taskresult_48 on host3:48924 in memory (size: 11.4 MB, free: 3.6 GB)
15/08/31 08:29:51 INFO BlockManagerInfo: Added taskresult_50 in memory on host3:48924 (size: 11.6 MB, free: 3.6 GB)
15/08/31 08:29:52 INFO BlockManagerInfo: Added taskresult_51 in memory on host2:60182 (size: 11.6 MB, free: 3.6 GB)
15/08/31 08:30:02 ERROR Utils: Uncaught exception in thread task-result-getter-0
java.lang.OutOfMemoryError: GC overhead limit exceeded
            at java.util.Arrays.copyOfRange(Arrays.java:2694)
            at java.lang.String.<init>(String.java:203)
            at com.esotericsoftware.kryo.io.Input.readString(Input.java:448)
            at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:157)
            at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:146)
            at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
            at com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:42)
            at com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:33)
            at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
            at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:338)
            at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:293)
            at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
            at org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:173)
            at org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:79)
            at org.apache.spark.scheduler.TaskSetManager.handleSuccessfulTask(TaskSetManager.scala:621)
            at org.apache.spark.scheduler.TaskSchedulerImpl.handleSuccessfulTask(TaskSchedulerImpl.scala:379)
            at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:82)
            at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:51)
            at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:51)
            at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
            at org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:50)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

Answer 1

从日志看来，驱动程序内存不足。

对于像 collect 这样的某些操作，所有工作人员的 rdd 数据都会传输到驱动程序 JVM。

1. 检查驱动程序 JVM 设置
2. 避免在驱动程序 JVM 上收集太多数据