Spark CassandraTableScanRDD KeyBy 不保留所有列

Question

CASSANDRA_TABLE has (some_other_column, itemid) as primary key.

val cassandraRdd: CassandraTableScanRDD[CassandraRow] = sparkSession.sparkContext
  .cassandraTable(cassandraKeyspace, cassandraTable)

cassandraRdd.take(10).foreach(println)

这个 cassandraRdd 包含从我的 cassandra 表中读取的所有列

val temp1: CassandraTableScanRDD[((String), CassandraRow)] = cassandraRdd
  .select("itemid", "column2", "column3")
  .keyBy[(String)]("itemid")
val temp2: CassandraTableScanRDD[((String), CassandraRow)] = cassandraRdd
  .keyBy[(String)]("itemid")
temp1.take(10).foreach(println)
temp2.take(10).foreach(println)

在 keyBy 操作之后，temp1 和 temp2 都没有保留所有列

((988230014),CassandraRow{itemid: 988230014})

如何在某些列上进行 keyBy 并让 CassandraRow 保留所有列？

Answer 1

要保留分区程序并获取选定的行，我必须阅读下面这样的 cassandra 行

val cassandraRdd: CassandraTableScanRDD[((String, String), (String, String, String))] = {
  sparkSession.sparkContext
    .cassandraTable[(String, String, String)](cassandraKeyspace, cassandraTable)
    .select("some_other_column" as "_1", "itemid" as "_2", "column3" as "_3", "some_other_column", "itemid")
    .keyBy[(String, String)]("some_other_column", "itemid")
}