SparkStreaming on Kafka Direct 提交 yarn

前言：我们将写好的 SparkStreaming 消费 kafka 的代码提交到 yarn 上运行

1. spark-submit
   –name StreamingKafkaYARNApp
   –master yarn
   –deploy-mode client
   –class com.ruozedata.bigdata.spark.streaming04.StreamingKafkaYARNApp
   /home/hadoop/lib/ruozedata-spark-1.0.jar
   spark000:9092,spark000:9093,spark000:9094 ruozedata_offset ruozedata_groupid_wuji
   
   直接提交会报以上错误：
   Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/StringDeserializer

   看官网的描述：
   For Scala and Java applications, if you are using SBT or Maven for project management, then package spark-streaming- kafka-0-10_2.12 and its dependencies into the application JAR.

2. spark-submit
   –name StreamingKafkaYARNApp
   –master yarn
   –deploy-mode client
   –packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.4.4
   –class com.ruozedata.bigdata.spark.streaming04.StreamingKafkaYARNApp
   /home/hadoop/lib/ruozedata-spark-1.0.jar
   spark000:9092,spark000:9093,spark000:9094 ruozedata_offset ruozedata_groupid_wuji

   我们看下 spark-submit --help 的命令帮助，发现：
   –packages
   Comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. Will search the local maven repo, then maven central and any additional remote repositories given by --repositories. The format for the coordinates should be groupId:artifactId:version.
   
   我们看到：它会去下载这个依赖，不过这要求服务器能连外网

   下载完后，程序正常运行

3. spark-submit
   –name StreamingKafkaYARNApp
   –master yarn
   –deploy-mode client
   –jars /home/hadoop/lib/spark-streaming-kafka-0-10_2.11-2.4.4.jar,/home/hadoop/lib/kafka-clients-2.0.0.jar
   –class com.ruozedata.bigdata.spark.streaming04.StreamingKafkaYARNApp
   /home/hadoop/lib/ruozedata-spark-1.0.jar
   spark000:9092,spark000:9093,spark000:9094 ruozedata_offset ruozedata_groupid_wuji

   采用 --jars 的方式，将所需jar包放到指定目录下

   程序正常运行