过滤火花CassandaRow RDD的每一行[重复]

Question

我有以下代码：-

 val rss = sc.cassandraTable("db", "table").select("id", "date", "gpsdt").where("id=? and date=? and gpsdt>? and gpsdt<?", entry(0), entry(1), entry(2) , entry(3))

    rss.foreach { records =>
      {
        println("Cassandra Row " + records.toString())
        val gpsdttime = records.get[String]("gpsdt")
        val justLess = rss.filter(row => row.get[String]("gpsdt") < gpsdttime).sortBy(row => row.get[String]("gpsdt"), false).take(1)
      }
    }

所以，我的想法是根据一些 where 子句从 Cassandra 中选择一组 RDD，并遍历每一行并找到其各自的前一行以重新计算一些值并更新当前行。但这给出了一个错误：-

org.apache.spark.SparkException: This RDD lacks a SparkContext. It could happen in the following cases: 
(1) RDD transformations and actions are NOT invoked by the driver, but inside of other transformations; for example, rdd1.map(x => rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063.
(2) When a Spark Streaming job recovers from checkpoint, this exception will be hit if a reference to an RDD not defined by the streaming job is used in DStream operations. For more information, See SPARK-13758.
at org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$sc(RDD.scala:89) 

请提出建议，谢谢，

Answer 1

Exception的意思是SparkContext在driver中初始化，而foreach中的func在executor中运行，所以当你运行job时，它会抛出 以下异常：

org.apache.spark.SparkException: This RDD lacks a SparkContext.

您的情况的原因是 (1) RDD 转换和操作不是由驱动程序调用的，而是在其他转换内部；例如，rdd1.map(x => rdd2.values.count() * x) 无效，因为值转换和计数操作无法在 rdd1.map 转换内部执行。有关详细信息，请参阅 SPARK-5063。