aws emr pyspark 卡在对方付费电话上答案

【问题标题】：aws emr pyspark stuck on collect callaws emr pyspark 卡在对方付费电话上
【发布时间】：2021-01-15 01:48:57
【问题描述】：

我正在尝试学习在 aws emr 上设置 pyspark。但是，我正在运行的示例作业卡在 collect api 调用中。我正在使用 emr 版本 5-30.1。我没有看到与此相关的任何相关日志。可能是，我没找对地方。

下面是 spark-submit 命令

spark-submit --deploy-mode cluster --master yarn --conf spark.yarn.maxAppAttempts=1 --conf spark.driver.memory=4g --conf spark.dynamicAllocation.enabled=true --conf spark.driver.cores=2 --conf spark.logConf=true --py-files s3://cdk_assets/assets/cdk.out/asset.5a391.zip s3://cdk_assets/assets/cdk.out/asset.3ab5bae.py'

这是我正在运行的示例脚本

from pyspark.sql import SparkSession

def run():
    print("Sample started")
    spark_session = SparkSession.builder \
        .appName("Spark_Sample") \
        .master("yarn") \
        .getOrCreate()
    temp_c = [10, 3, -5, 25, 1, 9, 29, -10, 5]
    rdd_temp_c = spark_session.sparkContext.parallelize(temp_c)
    rdd_temp_k = rdd_temp_c.map(lambda x: x + 273.15).collect()
    print(rdd_temp_k)

if __name__ == "__main__":
    run()

我可以看到以下异常，但我不确定它们之间的关系。第一个在node//provision-node/apps-phase//install.stderr下

2020-09-28 15:30:37,496 ERROR main: Encountered a problem while provisioning
java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:607)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
    at sun.net.www.http.HttpClient.New(HttpClient.java:339)
    at sun.net.www.http.HttpClient.New(HttpClient.java:357)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
    at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
    at com.amazonaws.emr.node.provisioner.http.JsonHttpClient.doRequest(JsonHttpClient.java:49)
    at com.amazonaws.emr.node.provisioner.platform.EmrPlatformClient.getConfiguration(EmrPlatformClient.java:51)
    at com.amazonaws.emr.node.provisioner.platform.EmrPlatformClient.getConfiguration(EmrPlatformClient.java:43)
    at com.amazonaws.emr.node.provisioner.bigtop.config.PlatformContextProvider.provide(PlatformContextProvider.java:39)
    at com.amazonaws.emr.node.provisioner.workflow.NodeProvisionerWorkflow.work(NodeProvisionerWorkflow.java:97)
    at com.amazonaws.emr.node.provisioner.Program.main(Program.java:30)

第二个来自yarn node-manager日志

2020-09-28 15:34:37,249 INFO org.apache.hadoop.ipc.Client (main): Retrying connect to server: ip-10-0-217-37.us-west-2.compute.internal/10.0.217.37:8025. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-09-28 15:34:38,250 INFO org.apache.hadoop.ipc.Client (main): Retrying connect to server: ip-10-0-217-37.us-west-2.compute.internal/10.0.217.37:8025. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-09-28 15:34:39,251 INFO org.apache.hadoop.ipc.Client (main): Retrying connect to server: ip-10-0-217-37.us-west-2.compute.internal/10.0.217.37:8025. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-09-28 15:34:39,252 WARN org.apache.hadoop.ipc.Client (main): Failed to connect to server: ip-10-0-217-37.us-west-2.compute.internal/10.0.217.37:8025: retries get failed due to exceeded maximum allowed retries number: 10
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:685)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:788)
    at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:410)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1550)
    at org.apache.hadoop.ipc.Client.call(Client.java:1381)
    at org.apache.hadoop.ipc.Client.call(Client.java:1345)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
    at com.sun.proxy.$Proxy79.registerNodeManager(Unknown Source)
    at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:73)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
    at com.sun.proxy.$Proxy80.registerNodeManager(Unknown Source)
    at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:344)
    at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:232)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
    at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:369)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:637)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:684)
2020-09-28 15:35:09,840 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager (main): Rolling master-key for container-tokens, got key with id 1723617652
2020-09-28 15:35:09,845 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM (main): Rolling master-key for container-tokens, got key with id 1930837132
2020-09-28 15:35:09,845 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl (main): NodeLabels sent from NM while registration were rejected by RM. And with message Node Labels {CORE} reported from NM with ID ip-10-0-192-171.us-west-2.compute.internal:8041 was rejected from RM with exception message as : Not all labels being replaced contained by known label collections, please check, new labels=[CORE]
2020-09-28 15:35:09,845 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl (main): Registered with ResourceManager as ip-10-0-192-171.us-west-2.compute.internal:8041 with total resource of <memory:6144, vCores:4>
2020-09-28 15:35:09,846 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl (main): Notifying ContainerManager to unblock new container-requests
2020-09-28 15:35:09,938 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl (Node Status Updater): NM node labels {<CORE:exclusivity=true>} were not accepted by RM and message from RM : Node Labels {CORE} reported from NM with ID ip-10-0-192-171.us-west-2.compute.internal:8041 was rejected from RM with exception message as : Not all labels being replaced contained by known label collections, please check, new labels=[CORE]

有人可以指导我如何进一步调试此问题。提前致谢。

【问题讨论】：

您可以进行以下更改并告诉我会发生什么吗？如果可行，我会将其转换为答案。将其用于会话构建` spark_session = SparkSession\ .builder\ .appName("spark sample)\ .getOrCreate()` 并使用 `spark-submit --py-files s3://cdk_assets/assets/cdk.out/ 提交asset.5a391.zip s3://cdk_assets/assets/cdk.out/asset.3ab5bae.py'
Emr 我猜默认使用 Yarn，另外，你真的需要所有这些配置，就像你提供的提交命令一样吗？
@A.B 确实有效。在我的反复试验中，我发现与众不同的是部署模式。当我将它从集群更改为客户端时，它开始工作。但我不确定为什么会这样。你知道原因吗？
我已转换此信息答案并尝试回答 ypur 查询，请随时投票并接受答案
另外，如果您的工作节点或驱动程序/主节点的 IP 为 10.0.217.37，您能否告诉我。它属于哪个节点？

标签： apache-spark hadoop pyspark hadoop-yarn amazon-emr

【解决方案1】：

Emr默认使用Yarn我猜，你可以先尝试简单的会话配置和spark-submit命令排除其他问题吗？

将此用于会话构建

 spark_session = SparkSession\ .builder\ .appName("spark sample)\ .getOrCreate()

并提交

  spark-submit --py-files s3://cdk_assets/assets/cdk.out/asset.5a391.zip s3://cdk_assets/assets/cdk.out/asset.3ab5bae.py

更新：为什么它以客户端的部署模式运行

默认情况下，spark 应用程序以客户端模式运行，即驱动程序在您提交应用程序的节点上运行。

同样，如果你使用集群模式，他 spark 作业的“驱动程序”组件不会在提交作业的本地计算机上运行。例如，它将在工人上运行，因此您需要从那里与 master(yarn) 建立正确的连接。

正如我在评论中提到的，请确保您确实需要使用 spark 提交命令提供的所有配置，如果没有，请使用所需参数提交。

【讨论】：