为 EMR 上的“过度”并行性调整 Spark

Question

我有一个 Spark 作业，它读取一些 TB 的数据并执行两个窗口函数。这项工作在较小的块中运行得很好，4TB 上的 50k shuffle 分区，但是当我将数据输入增加到 150k-200k 时，15TB 节点的 shuffle 分区开始失败。

发生这种情况有两个原因：

• 执行器上的 OOM：
• 洗牌时超时

执行者的OOM

20/07/01 15:58:14 ERROR YarnClusterScheduler: Lost executor 92 on ip-10-102-125-133.ec2.internal: Container killed by YARN for exceeding memory limits.  22.0 GB of 22 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714.

我已经增加了驱动程序的大小以适应大洗牌：

• spark.driver.memory = 16g
• spark.driver.maxResultSize = 8g

执行器是 R5.xlarge，配置如下：

• spark.executor.cores = 4
• spark.executor.memory = 18971M
• spark.yarn.executor.memoryOverheadFactor = 0.1875

这远低于 AWS 规定的最大值：https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-task-config.html#emr-hadoop-task-config-r5

• yarn.nodemanager.resource.memory-mb = 24576

我知道我需要在此处调整 spark.yarn.executor.memoryOverheadFactor 以便为与这么多分区相关的大量开销留出空间。希望这将是那里需要的最后一次更改。

随机播放超时

20/07/01 15:59:39 ERROR TransportChannelHandler: Connection to ip-10-102-116-184.ec2.internal/10.102.116.184:7337 has been quiet for 600000 ms while there are outstanding requests. Assuming connection is dead; please adjust spark.network.timeout if this is wrong.
20/07/01 15:59:39 ERROR TransportResponseHandler: Still have 8 requests outstanding when connection from ip-10-102-116-184.ec2.internal/10.102.116.184:7337 is closed
20/07/01 15:59:39 ERROR OneForOneBlockFetcher: Failed while starting block fetches

我已将此超时调整如下：

• spark.network.timeout = 600

我可以进一步增加 conf 中的 spark.network.timeout 以使其安静并等待更长时间。我宁愿减少Shuffle Read Blocked Time，这是从1分钟到30分钟不等。有没有办法提高节点间的通信速率？

我已尝试调整以下设置，但似乎无法提高此速度：

• spark.reducer.maxSizeInFlight = 512m
• spark.shuffle.io.numConnectionsPerPeer = 5
• spark.shuffle.io.backLog = 128

我需要调整什么来减少 AWS EMR 上的 Shuffle Read Blocked Time？

Answer 1

对于 executors 上的 OOM，请执行此操作。它为我们解决了这个问题。 来自：https://aws.amazon.com/blogs/big-data/best-practices-for-successfully-managing-memory-for-apache-spark-applications-on-amazon-emr/

Even if all the Spark configuration properties are calculated and set correctly, virtual out-of-memory errors can still occur rarely as virtual memory is bumped up aggressively by the OS. To prevent these application failures, set the following flags in the YARN site settings.

Best practice 5: Always set the virtual and physical memory check flag to false.

"yarn.nodemanager.vmem-check-enabled":"false",
"yarn.nodemanager.pmem-check-enabled":"false"

原因："Container killed by YARN for exceeding memory limits. 10.4 GB of 10.4 GB physical memory used" on an EMR cluster with 75GB of memory

要解决 shuffle-timeout，请尝试增加您的存储（EBS 卷）。