向 Spark 提交任务

Question

我按照本教程在 ubuntu 14.04 上安装了 spark http://blog.prabeeshk.com/blog/2014/10/31/install-apache-spark-on-ubuntu-14-dot-04/

我能够运行 spark 中提供的示例，并且它似乎可以工作。

问题是我无法制作 scala 文件并用 spark 执行它。这是我按照https://spark.apache.org/docs/latest/quick-start.html

的指导方针所做的

我的独立应用是：

/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.commons.math3.random.RandomDataGenerator


object SimpleApp {
  def main(args: Array[String]) {
    val logFile = "/home/donbeo/Applications/spark/spark-1.1.0/README.md" // Should be some file on your system
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    val numBs = logData.filter(line => line.contains("b")).count()
    println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))


    println("A random number")

  val randomData = new RandomDataGenerator()

  println(randomData.nextLong(0, 100))
  }
}

我的 sbt 文件是：

   name := "Simple Project"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0"

libraryDependencies += "org.apache.commons" % "commons-math3" % "3.3"

我的项目结构是：

donbeo@donbeo-HP-EliteBook-Folio-9470m:~/Documents/scala_code/simpleApp$ find .
.
./src
./src/main
./src/main/scala
./src/main/scala/SimpleApp.scala~
./src/main/scala/SimpleApp.scala
./simple.sbt
donbeo@donbeo-HP-EliteBook-Folio-9470m:~/Documents/scala_code/simpleApp$ 

然后我跑

donbeo@donbeo-HP-EliteBook-Folio-9470m:~/Documents/scala_code/simpleApp$ sbt package
[info] Set current project to Simple Project (in build file:/home/donbeo/Documents/scala_code/simpleApp/)
[info] Updating {file:/home/donbeo/Documents/scala_code/simpleApp/}simpleapp...
[info] Resolving org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645 .[info] Resolving org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v20100508202[info] Resolving org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233 ..[info] Resolving org.spark-project.akka#akka-remote_2.10;2.2.3-shaded-protobuf .[info] Resolving org.spark-project.akka#akka-actor_2.10;2.2.3-shaded-protobuf ..[info] Resolving org.spark-project.akka#akka-slf4j_2.10;2.2.3-shaded-protobuf ..[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 1 Scala source to /home/donbeo/Documents/scala_code/simpleApp/target/scala-2.10/classes...
[info] Packaging /home/donbeo/Documents/scala_code/simpleApp/target/scala-2.10/simple-project_2.10-1.0.jar ...
[info] Done packaging.
[success] Total time: 8 s, completed 04-Feb-2015 15:20:09
donbeo@donbeo-HP-EliteBook-Folio-9470m:~/Documents/scala_code/simpleApp$ 

在最后一步我得到一个错误

donbeo@donbeo-HP-EliteBook-Folio-9470m:~/Applications/spark/spark-1.1.0$ ./bin/spark-submit \ --class "SimpleApp" \ --master local[4] \ /home/donbeo/Documents/scala_code/simpleApp/target/scala-2.10/simple-project_2.10-1.0.jar
Exception in thread "main" java.net.URISyntaxException: Illegal character in path at index 0:  --class
    at java.net.URI$Parser.fail(URI.java:2829)
    at java.net.URI$Parser.checkChars(URI.java:3002)
    at java.net.URI$Parser.parseHierarchical(URI.java:3086)
    at java.net.URI$Parser.parse(URI.java:3044)
    at java.net.URI.<init>(URI.java:595)
    at org.apache.spark.util.Utils$.resolveURI(Utils.scala:1343)
    at org.apache.spark.deploy.SparkSubmitArguments.parse$1(SparkSubmitArguments.scala:338)
    at org.apache.spark.deploy.SparkSubmitArguments.parseOpts(SparkSubmitArguments.scala:225)
    at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:60)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:70)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
donbeo@donbeo-HP-EliteBook-Folio-9470m:~/Applications/spark/spark-1.1.0$ 

我做错了吗？我该如何解决？

Answer 1

您需要从他们的命令行示例中删除所有\，因为换行符已被添加：

./bin/spark-submit --class "SimpleApp" --master local[4] /home/donbeo/Documents/scala_code/simpleApp/target/scala-2.10/simple-project_2.10-1.0.jar