【发布时间】：2019-11-29 08:53:46
【问题描述】：
我正在运行一个 Spark 作业，如果我对样本数据执行计算（想想大约 1000 行），那么一切正常。但是当我尝试在更大的数据集上执行相同的计算时，我得到了
19/07/20 14:21:53 WARN TaskSetManager: Lost task 198.0 in stage 150.0 (TID 21928, 10.46.225.176, executor 17): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/databricks/spark/python/pyspark/worker.py", line 403, in main
process()
File "/databricks/spark/python/pyspark/worker.py", line 398, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/databricks/spark/python/pyspark/rdd.py", line 2516, in pipeline_func
return func(split, prev_func(split, iterator))
File "/databricks/spark/python/pyspark/rdd.py", line 2516, in pipeline_func
return func(split, prev_func(split, iterator))
File "/databricks/spark/python/pyspark/rdd.py", line 352, in func
return f(iterator)
File "/databricks/spark/python/pyspark/rdd.py", line 1886, in _mergeCombiners
merger.mergeCombiners(iterator)
File "/databricks/spark/python/pyspark/shuffle.py", line 289, in mergeCombiners
self._spill()
File "/databricks/spark/python/pyspark/shuffle.py", line 317, in _spill
self.serializer.dump_stream([(k, v)], streams[h])
File "/databricks/spark/python/pyspark/serializers.py", line 417, in dump_stream
bytes = self.serializer.dumps(vs)
File "/databricks/spark/python/pyspark/serializers.py", line 679, in dumps
return zlib.compress(self.serializer.dumps(obj), 1)
OverflowError: size does not fit in an int
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:490)
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:626)
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:609)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:444)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:349)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1187)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1161)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1096)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1161)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:883)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:351)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:302)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:75)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:340)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:304)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.doRunTask(Task.scala:139)
at org.apache.spark.scheduler.Task.run(Task.scala:112)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1481)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
是什么触发了它？我想在收集操作的最后阶段有些事情
rdd.take(500)
到目前为止，我已经尝试过：
- 重新分区到 4000 个分区。没有帮助/我无法理解是否有帮助。
- 使用大型集群 - m5.xlarge + r4.4xlarge（16 个工作器）。使用较小的集群会对此有所帮助。大型集群是否可能会导致一些序列化问题？
- 使用 Python 2.7，因为我使用的库是用 2.7 编写的。我看到一篇帖子说 zlib 可能存在问题，但我不确定如何修复它或解决它。
我觉得我已经用尽了我对这个问题的所有有限理解。非常感谢任何可能有用的指导或事情。请不要标记它是重复的，因为我检查了它周围的几个帖子，并没有发现任何有用的东西。
【问题讨论】：
-
对 python2 的 spark 支持即将结束。 link
标签： python scala apache-spark pyspark apache-spark-sql