用于 Twitter 流的 Spark 执行

Question

您好，我是 spark 和 scala 的新手。我正在尝试使用以下代码通过火花流传输一些推文：

object TwitterStreaming {

  def main(args: Array[String]): Unit = {
    if (args.length < 1) {
      System.err.println("WrongUsage:   PropertiesFile, [<filters>]")
      System.exit(-1)
    }

    StreamingExamples.setStreaningLogLevels()
    val myConfigFile = args(0)
    val batchInterval_s = 1
    val fileConfig = ConfigFactory.parseFile(new File(myConfigFile))
    val appConf = ConfigFactory.load(fileConfig)  
    // Set the system properties so that Twitter4j library used by twitter stream
    // can use them to generate OAuth credentials

    System.setProperty("twitter4j.oauth.consumerKey", appConf.getString("consumerKey"))
    System.setProperty("twitter4j.oauth.consumerSecret", appConf.getString("consumerSecret"))
    System.setProperty("twitter4j.oauth.accessToken", appConf.getString("accessToken"))
    System.setProperty("twitter4j.oauth.accessTokenSecret", appConf.getString("accessTokenSecret"))

    val sparkConf = new SparkConf().setAppName("TwitterStreaming").setMaster(appConf.getString("SPARK_MASTER"))//local[2]

    val ssc = new StreamingContext(sparkConf, Seconds(batchInterval_s)) // creating spark streaming context
    val stream = TwitterUtils.createStream(ssc, None)
    val tweet_data = stream.map(status => TweetData(status.getId, "@" + status.getUser.getScreenName, status.getText.trim()))
    tweet_data.foreachRDD(rdd => {
      println(s"A sample of tweets I gathered over ${batchInterval_s}s: ${rdd.take(10).mkString(" ")} (total tweets fetched: ${rdd.count()})")
    })
  }

}

case class TweetData(id: BigInt, author: String, tweetText: String)

我的错误：

Exception in thread "main" com.typesafe.config.ConfigException$WrongType:/WorkSpace/InputFiles/application.conf: 5: Cannot concatenate object or list with a non-object-or-list, ConfigString("local") and SimpleConfigList([2]) are not compatible
at com.typesafe.config.impl.ConfigConcatenation.join(ConfigConcatenation.java:116)

谁能检查代码并告诉我哪里做错了？

Answer 1

如果您的配置文件包含：

SPARK_MASTER=local[2]

改成：

SPARK_MASTER="local[2]"