【发布时间】：2016-02-01 13:09:52
【问题描述】：
我正在尝试运行一个用 python 编写的简单 Spark 流式传输作业：
#!/usr/bin/env python
from pyspark import SparkContext, SparkConf
from pyspark.streaming import StreamingContext
conf = SparkConf()
conf.setMaster("spark://master1:7077,master2:7077")
sc = SparkContext(conf=conf)
ssc = StreamingContext(sc, 1)
ssc.socketTextStream("master1", 9999).count().pprint()
ssc.start()
ssc.awaitTermination()
运行几秒钟后，任务失败。这是我看到的异常：
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at com.esotericsoftware.kryo.io.Output.flush(Output.java:155)
at com.esotericsoftware.kryo.io.Output.require(Output.java:135)
at com.esotericsoftware.kryo.io.Output.writeString_slow(Output.java:420)
at com.esotericsoftware.kryo.io.Output.writeString(Output.java:326)
at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:153)
at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:146)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:158)
at org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:153)
at org.apache.spark.storage.BlockManager.dataSerializeStream(BlockManager.scala:1190)
at org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1199)
at org.apache.spark.storage.MemoryStore.putArray(MemoryStore.scala:132)
at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:169)
at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:143)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:791)
at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:638)
at org.apache.spark.streaming.receiver.BlockManagerBasedBlockHandler.storeBlock(ReceivedBlockHandler.scala:77)
at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushAndReportBlock(ReceiverSupervisorImpl.scala:156)
at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushArrayBuffer(ReceiverSupervisorImpl.scala:127)
at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl$$anon$3.onPushBlock(ReceiverSupervisorImpl.scala:108)
at org.apache.spark.streaming.receiver.BlockGenerator.pushBlock(BlockGenerator.scala:294)
at org.apache.spark.streaming.receiver.BlockGenerator.org$apache$spark$streaming$receiver$BlockGenerator$$keepPushingBlocks(BlockGenerator.scala:266)
at org.apache.spark.streaming.receiver.BlockGenerator$$anon$1.run(BlockGenerator.scala:108)
之后会启动一个新任务，因此该作业会继续运行。但是，我想知道，我错过了什么。
更新
spark-defaults.conf
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.driver.memory 4g
spark.executor.memory 4g
spark.executor.extraJavaOptions -XX:+PrintGCDetails
spark.deploy.recoveryMode ZOOKEEPER
spark.deploy.zookeeper.url master1:2181,master2:2181,master3:2181
【问题讨论】：
-
你在哪里看到的？驱动程序还是执行程序？似乎您需要增加执行者的内存。另请提及您的集群配置。
-
我在执行程序中看到了这个异常。每个执行器都有 4Gb 的 RAM。我已经更新了发布我的 spark-defaults.conf 的问题
-
每批流中接收到的数据的大小/类型是多少？如果您已经捕获了 GC 日志，也可以发布该日志。您的程序很简单，但似乎接收数据的数据速度太高。您是否在 Spark-UI 中看到任何内容，例如任务 Backlog 等。
-
感谢您的回复。我是新来的火花，不知道在任务积压中寻找什么。该程序正在从网络套接字中提取大约 1Kb 的字符串。在收到大约 1-3kk 的字符串后，我看到执行程序以“内存不足”退出。
-
我还尝试将执行者的内存加倍。 “内存不足”也会发生。
标签： python apache-spark spark-streaming