在本地集群上运行 spark 作业无限运行

【问题标题】：running spark job on local cluster runs infinitely在本地集群上运行 spark 作业无限运行
【发布时间】：2019-05-30 00:38:09
【问题描述】：

我已经在我的 Windows 7 机器（一个主节点和工作节点）上设置了一个本地 Spark 集群。我创建了一个简单的 scala 脚本，我使用 sbt 构建并尝试使用 spark-submit 运行。请在下面找到资源

Scala 代码：

package example1

import java.io._

import org.apache.spark.sql.SQLContext 
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.functions.expr
import org.apache.spark.SparkContext
import org.apache.spark.sql.SparkSession

object HelloWorld {

    def main(args: Array[String]): Unit = {
        println("===============================================")
        println("===============================================") 
        println("Hello, world!")
        val pw = new PrintWriter(new File("d:\\hello.txt" ))
        pw.write("Hello, world")

        println("===============================================")
        println("===============================================")

        val session = SparkSession.builder.getOrCreate()


        var filesmall = "file:///D:/_Work/azurepoc/samplebigdata/ds2.csv"

        //val df  =  session.read.format("csv").option("header", "true").load(filesmall)

        println("===============================================")

        pw.write("Hello, world some more information ")
        pw.close
    }
}

Spark 集群主脚本：

C:\Windows\system32>spark-class org.apache.spark.deploy.master.Master
2019-01-03 16:49:16 INFO  Master:2612 - Started daemon with process name: 23940@ws-amalhotra
2019-01-03 16:49:16 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-01-03 16:49:16 INFO  SecurityManager:54 - Changing view acls to: admin
2019-01-03 16:49:16 INFO  SecurityManager:54 - Changing modify acls to: admin
2019-01-03 16:49:16 INFO  SecurityManager:54 - Changing view acls groups to:
2019-01-03 16:49:16 INFO  SecurityManager:54 - Changing modify acls groups to:
2019-01-03 16:49:16 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(admin); groups with view permissions: Set(); user
s  with modify permissions: Set(admin); groups with modify permissions: Set()
2019-01-03 16:49:17 INFO  Utils:54 - Successfully started service 'sparkMaster' on port 7077.
2019-01-03 16:49:17 INFO  Master:54 - Starting Spark master at spark://192.168.8.101:7077
2019-01-03 16:49:17 INFO  Master:54 - Running Spark version 2.3.2
2019-01-03 16:49:17 INFO  log:192 - Logging initialized @1412ms
2019-01-03 16:49:17 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2019-01-03 16:49:17 INFO  Server:419 - Started @1489ms
2019-01-03 16:49:17 INFO  AbstractConnector:278 - Started ServerConnector@16391414{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2019-01-03 16:49:17 INFO  Utils:54 - Successfully started service 'MasterUI' on port 8080.
2019-01-03 16:49:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@204e3825{/app,null,AVAILABLE,@Spark}
2019-01-03 16:49:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@748394e8{/app/json,null,AVAILABLE,@Spark}
2019-01-03 16:49:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@19b99890{/,null,AVAILABLE,@Spark}
2019-01-03 16:49:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5c0f561c{/json,null,AVAILABLE,@Spark}
2019-01-03 16:49:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3443bda1{/static,null,AVAILABLE,@Spark}
2019-01-03 16:49:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54541f46{/app/kill,null,AVAILABLE,@Spark}
2019-01-03 16:49:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6e8c3d12{/driver/kill,null,AVAILABLE,@Spark}
2019-01-03 16:49:17 INFO  MasterWebUI:54 - Bound MasterWebUI to 0.0.0.0, and started at http://ws-amalhotra.domain.co.in:8080
2019-01-03 16:49:17 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2019-01-03 16:49:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@22eb9260{/,null,AVAILABLE}
2019-01-03 16:49:17 INFO  AbstractConnector:278 - Started ServerConnector@636eb125{HTTP/1.1,[http/1.1]}{192.168.8.101:6066}
2019-01-03 16:49:17 INFO  Server:419 - Started @1558ms
2019-01-03 16:49:17 INFO  Utils:54 - Successfully started service on port 6066.
2019-01-03 16:49:17 INFO  StandaloneRestServer:54 - Started REST server for submitting applications on port 6066
2019-01-03 16:49:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1a4c3e84{/metrics/master/json,null,AVAILABLE,@Spark}
2019-01-03 16:49:17 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5a3b4746{/metrics/applications/json,null,AVAILABLE,@Spark}
2019-01-03 16:49:17 INFO  Master:54 - I have been elected leader! New state: ALIVE
2019-01-03 16:49:21 INFO  Master:54 - Registering worker 192.168.8.101:8089 with 8 cores, 14.9 GB RAM

我的工作节点：

C:\Windows\system32>spark-class org.apache.spark.deploy.worker.Worker spark://192.168.8.101:7077 -p 8089
2019-01-03 16:49:20 INFO  Worker:2612 - Started daemon with process name: 16264@ws-amalhotra
2019-01-03 16:49:21 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-01-03 16:49:21 INFO  SecurityManager:54 - Changing view acls to: admin
2019-01-03 16:49:21 INFO  SecurityManager:54 - Changing modify acls to: admin
2019-01-03 16:49:21 INFO  SecurityManager:54 - Changing view acls groups to:
2019-01-03 16:49:21 INFO  SecurityManager:54 - Changing modify acls groups to:
2019-01-03 16:49:21 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(admin); groups with view permissions: Set(); user
s  with modify permissions: Set(admin); groups with modify permissions: Set()
2019-01-03 16:49:21 INFO  Utils:54 - Successfully started service 'sparkWorker' on port 8089.
2019-01-03 16:49:21 INFO  Worker:54 - Starting Spark worker 192.168.8.101:8089 with 8 cores, 14.9 GB RAM
2019-01-03 16:49:21 INFO  Worker:54 - Running Spark version 2.3.2
2019-01-03 16:49:21 INFO  Worker:54 - Spark home: C:\spark
2019-01-03 16:49:21 INFO  log:192 - Logging initialized @1471ms
2019-01-03 16:49:21 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2019-01-03 16:49:21 INFO  Server:419 - Started @1518ms
2019-01-03 16:49:21 INFO  AbstractConnector:278 - Started ServerConnector@44629c8f{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}
2019-01-03 16:49:21 INFO  Utils:54 - Successfully started service 'WorkerUI' on port 8081.
2019-01-03 16:49:21 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@36f34cce{/logPage,null,AVAILABLE,@Spark}
2019-01-03 16:49:21 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@447fb46{/logPage/json,null,AVAILABLE,@Spark}
2019-01-03 16:49:21 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3b027ba{/,null,AVAILABLE,@Spark}
2019-01-03 16:49:21 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5396b0bb{/json,null,AVAILABLE,@Spark}
2019-01-03 16:49:21 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6830ec44{/static,null,AVAILABLE,@Spark}
2019-01-03 16:49:21 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5eb28ff8{/log,null,AVAILABLE,@Spark}
2019-01-03 16:49:21 INFO  WorkerWebUI:54 - Bound WorkerWebUI to 0.0.0.0, and started at http://ws-amalhotra.domain.co.in:8081
2019-01-03 16:49:21 INFO  Worker:54 - Connecting to master 192.168.8.101:7077...
2019-01-03 16:49:21 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@36cc352{/metrics/json,null,AVAILABLE,@Spark}
2019-01-03 16:49:21 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.8.101:7077 after 26 ms (0 ms spent in bootstraps)
2019-01-03 16:49:21 INFO  Worker:54 - Successfully registered with master spark://192.168.8.101:7077

现在我使用 sbt 构建并打包 scala 代码，将其打包到 JAR 中。我的 build.sbt 文件如下所示 

version := "1.0" 
scalaVersion := "2.11.8" 
val sparkVersion = "2.0.0" 

libraryDependencies ++= Seq( 
    "org.apache.spark" %% "spark-core" % sparkVersion, 
    "org.apache.spark" %% "spark-streaming" % sparkVersion, 
    "org.apache.spark" %% "spark-sql" % sparkVersion 
    )

它会创建一个 jar，然后我使用 spark submit 命令提交它，如下所示：

C:\Users\amalhotra>spark-submit  --deploy-mode cluster --master spark://192.168.
8.101:6066 --class "example1.HelloWorld"  "D:\_Work\azurepoc\sbtexample\target\s
cala-2.11\sbtexample_2.11-1.0.jar"

一切正常，现在我只需在脚本中更改一行代码，然后再次按照 compile -> sbt package code -> spark-submit（同上）。代码更改是我取消注释以下行：

 //val df  =  session.read.format("csv").option("header", "true").load(filesmall)

当我再次使用 spark-submit 运行上述代码时，worker 将永远执行。另外，我的 D 盘中的文件没有被写入。工作人员日志如下

C:\Windows\system32>spark-class org.apache.spark.deploy.worker.Worker spark://192.168.8.101:7077 -p 8089
2019-01-03 17:24:38 INFO  Worker:2612 - Started daemon with process name: 24952@ws-amalhotra
2019-01-03 17:24:39 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-01-03 17:24:39 INFO  SecurityManager:54 - Changing view acls to: admin
2019-01-03 17:24:39 INFO  SecurityManager:54 - Changing modify acls to: admin
2019-01-03 17:24:39 INFO  SecurityManager:54 - Changing view acls groups to:
2019-01-03 17:24:39 INFO  SecurityManager:54 - Changing modify acls groups to:
2019-01-03 17:24:39 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(admin); groups with view permissions: Set(); user
s  with modify permissions: Set(admin); groups with modify permissions: Set()
2019-01-03 17:24:39 INFO  Utils:54 - Successfully started service 'sparkWorker' on port 8089.
2019-01-03 17:24:39 INFO  Worker:54 - Starting Spark worker 192.168.8.101:8089 with 8 cores, 14.9 GB RAM
2019-01-03 17:24:39 INFO  Worker:54 - Running Spark version 2.3.2
2019-01-03 17:24:39 INFO  Worker:54 - Spark home: C:\spark
2019-01-03 17:24:39 INFO  log:192 - Logging initialized @1512ms
2019-01-03 17:24:39 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2019-01-03 17:24:39 INFO  Server:419 - Started @1561ms
2019-01-03 17:24:39 INFO  AbstractConnector:278 - Started ServerConnector@51e2ccae{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}
2019-01-03 17:24:39 INFO  Utils:54 - Successfully started service 'WorkerUI' on port 8081.
2019-01-03 17:24:39 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3d96670b{/logPage,null,AVAILABLE,@Spark}
2019-01-03 17:24:39 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@48e02860{/logPage/json,null,AVAILABLE,@Spark}
2019-01-03 17:24:39 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@758918a3{/,null,AVAILABLE,@Spark}
2019-01-03 17:24:39 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1643bea5{/json,null,AVAILABLE,@Spark}
2019-01-03 17:24:39 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5f293725{/static,null,AVAILABLE,@Spark}
2019-01-03 17:24:39 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@339a8612{/log,null,AVAILABLE,@Spark}
2019-01-03 17:24:39 INFO  WorkerWebUI:54 - Bound WorkerWebUI to 0.0.0.0, and started at http://ws-amalhotra.domain.co.in:8081
2019-01-03 17:24:39 INFO  Worker:54 - Connecting to master 192.168.8.101:7077...
2019-01-03 17:24:39 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@196e9c2a{/metrics/json,null,AVAILABLE,@Spark}
2019-01-03 17:24:39 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.8.101:7077 after 29 ms (0 ms spent in bootstraps)
2019-01-03 17:24:40 INFO  Worker:54 - Successfully registered with master spark://192.168.8.101:7077
2019-01-03 17:25:17 INFO  Worker:54 - Asked to launch driver driver-20190103172517-0000
2019-01-03 17:25:17 INFO  DriverRunner:54 - Copying user jar file:/D:/_Work/azurepoc/sbtexample/target/scala-2.11/sbtexample_2.11-1.0.jar to C:\spark\work\driver-20190103172517-0000\sbtexamp
le_2.11-1.0.jar
2019-01-03 17:25:17 INFO  Utils:54 - Copying D:\_Work\azurepoc\sbtexample\target\scala-2.11\sbtexample_2.11-1.0.jar to C:\spark\work\driver-20190103172517-0000\sbtexample_2.11-1.0.jar
2019-01-03 17:25:17 INFO  DriverRunner:54 - Launch Command: "C:\Program Files\Java\jdk1.8.0_181\bin\java" "-cp" "C:\spark\bin\..\conf\;C:\spark\jars\*" "-Xmx1024M" "-Dspark.master=spark://19
2.168.8.101:7077" "-Dspark.driver.supervise=false" "-Dspark.submit.deployMode=cluster" "-Dspark.jars=file:/D:/_Work/azurepoc/sbtexample/target/scala-2.11/sbtexample_2.11-1.0.jar" "-Dspark.ap
p.name=example1.HelloWorld" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.8.101:8089" "C:\spark\work\driver-20190103172517-0000\sbtexample_2.11-1.0.jar" "example1.He
lloWorld"
2019-01-03 17:25:19 INFO  Worker:54 - Asked to launch executor app-20190103172519-0000/0 for example1.HelloWorld
2019-01-03 17:25:19 INFO  SecurityManager:54 - Changing view acls to: admin
2019-01-03 17:25:19 INFO  SecurityManager:54 - Changing modify acls to: admin
2019-01-03 17:25:19 INFO  SecurityManager:54 - Changing view acls groups to:
2019-01-03 17:25:19 INFO  SecurityManager:54 - Changing modify acls groups to:
2019-01-03 17:25:19 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(admin); groups with view permissions: Set(); user
s  with modify permissions: Set(admin); groups with modify permissions: Set()
2019-01-03 17:25:19 INFO  ExecutorRunner:54 - Launch command: "C:\Program Files\Java\jdk1.8.0_181\bin\java" "-cp" "C:\spark\bin\..\conf\;C:\spark\jars\*" "-Xmx1024M" "-Dspark.driver.port=557
86" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@ws-amalhotra.domain.co.in:55786" "--executor-id" "0" "--hostname" "192.168.8.101" "--
cores" "7" "--app-id" "app-20190103172519-0000" "--worker-url" "spark://Worker@192.168.8.101:8089"
2019-01-03 17:25:43 INFO  Worker:54 - Executor app-20190103172519-0000/0 finished with state EXITED message Command exited with code 1 exitStatus 1
2019-01-03 17:25:43 INFO  Worker:54 - Asked to launch executor app-20190103172519-0000/1 for example1.HelloWorld
2019-01-03 17:25:43 INFO  SecurityManager:54 - Changing view acls to: admin
2019-01-03 17:25:43 INFO  SecurityManager:54 - Changing modify acls to: admin
2019-01-03 17:25:43 INFO  SecurityManager:54 - Changing view acls groups to:
2019-01-03 17:25:43 INFO  SecurityManager:54 - Changing modify acls groups to:
2019-01-03 17:25:43 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(admin); groups with view permissions: Set(); user
s  with modify permissions: Set(admin); groups with modify permissions: Set()
2019-01-03 17:25:43 INFO  ExecutorRunner:54 - Launch command: "C:\Program Files\Java\jdk1.8.0_181\bin\java" "-cp" "C:\spark\bin\..\conf\;C:\spark\jars\*" "-Xmx1024M" "-Dspark.driver.port=557
86" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@ws-amalhotra.domain.co.in:55786" "--executor-id" "1" "--hostname" "192.168.8.101" "--
cores" "7" "--app-id" "app-20190103172519-0000" "--worker-url" "spark://Worker@192.168.8.101:8089"
2019-01-03 17:26:05 INFO  Worker:54 - Executor app-20190103172519-0000/1 finished with state EXITED message Command exited with code 1 exitStatus 1
2019-01-03 17:26:05 INFO  Worker:54 - Asked to launch executor app-20190103172519-0000/2 for example1.HelloWorld
2019-01-03 17:26:05 INFO  SecurityManager:54 - Changing view acls to: admin
2019-01-03 17:26:05 INFO  SecurityManager:54 - Changing modify acls to: admin
2019-01-03 17:26:05 INFO  SecurityManager:54 - Changing view acls groups to:
2019-01-03 17:26:05 INFO  SecurityManager:54 - Changing modify acls groups to:
2019-01-03 17:26:05 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(admin); groups with view permissions: Set(); user
s  with modify permissions: Set(admin); groups with modify permissions: Set()
2019-01-03 17:26:05 INFO  ExecutorRunner:54 - Launch command: "C:\Program Files\Java\jdk1.8.0_181\bin\java" "-cp" "C:\spark\bin\..\conf\;C:\spark\jars\*" "-Xmx1024M" "-Dspark.driver.port=557
86" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@ws-amalhotra.domain.co.in:55786" "--executor-id" "2" "--hostname" "192.168.8.101" "--
cores" "7" "--app-id" "app-20190103172519-0000" "--worker-url" "spark://Worker@192.168.8.101:8089"
2019-01-03 17:26:28 INFO  Worker:54 - Executor app-20190103172519-0000/2 finished with state EXITED message Command exited with code 1 exitStatus 1
2019-01-03 17:26:28 INFO  Worker:54 - Asked to launch executor app-20190103172519-0000/3 for example1.HelloWorld
2019-01-03 17:26:28 INFO  SecurityManager:54 - Changing view acls to: admin
2019-01-03 17:26:28 INFO  SecurityManager:54 - Changing modify acls to: admin
2019-01-03 17:26:28 INFO  SecurityManager:54 - Changing view acls groups to:
2019-01-03 17:26:28 INFO  SecurityManager:54 - Changing modify acls groups to:
2019-01-03 17:26:28 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(admin); groups with view permissions: Set(); user
s  with modify permissions: Set(admin); groups with modify permissions: Set()
2019-01-03 17:26:28 INFO  ExecutorRunner:54 - Launch command: "C:\Program Files\Java\jdk1.8.0_181\bin\java" "-cp" "C:\spark\bin\..\conf\;C:\spark\jars\*" "-Xmx1024M" "-Dspark.driver.port=557
86" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@ws-amalhotra.domain.co.in:55786" "--executor-id" "3" "--hostname" "192.168.8.101" "--
cores" "7" "--app-id" "app-20190103172519-0000" "--worker-url" "spark://Worker@192.168.8.101:8089"

这将永远运行，每隔几秒重复相同的日志。目前还不清楚发生了什么。日志没有说太多。没有完整的示例显示在本地独立集群上运行此类作业

【问题讨论】：

  • 看起来 spark 应用程序正在重试某事或等待某个事件。是不是因为你打开了文件，spark 正在等待锁被释放。
  • 不，我根本没有打开文件
  • 您是否尝试过完全删除PrintWriter 的使用，并从那里开始工作？
  • @AlexGrigore 相反，当我只保留与打印机相关的代码时，它可以完美运行。
  • @AlexGrigore 刚刚解决了上述问题。在我的个人机器上尝试了同样的方法，一切正常。这是与端口防火墙相关的问题。但我现在真的有一个严肃的问题。您一定已经观察到上述日志，我也这样做了... 为什么不清楚错误是什么？ 从日志中应该可以直观地看出我的防火墙设置有问题..可能是我做错了什么。我不太确定。

标签： scala apache-spark sbt spark-submit


【解决方案1】：

在 conf/spark-defaults.conf 中设置 spark.driver.host ！

【讨论】：

    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 2015-04-25
    • 1970-01-01
    • 2020-06-27
    • 2017-02-25
    • 1970-01-01
    • 1970-01-01
    • 2016-07-20
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode