writeStream() 正在批量打印空值数据，即使我通过 writeStream() 在 kafka 中提供了正确的 json 数据答案

【问题标题】：writeStream() is printing null values in batches data even i supply proper json data in kafka through writeStream()writeStream() 正在批量打印空值数据，即使我通过 writeStream() 在 kafka 中提供了正确的 json 数据
【发布时间】：2020-08-06 13:25:19
【问题描述】：

我正在尝试使用模式转换 json 并将值打印到控制台，但即使我提供了正确的数据，writeStream() 也会在所有列中打印空值。

我提供给 kafka 主题的数据 ..

{"stock":"SEE","buy":12,"sell":15,"profit":3,quantity:27,"loss":0,"gender":"M"}
{"stock":"SEE","buy":12,"sell":15,"profit":3,quantity:27,"loss":0,"gender":"M"}
{"stock":"SEE","buy":12,"sell":15,"profit":3,quantity:27,"loss":0,"gender":"M"}

下面是我的scala代码..

 val readStreamDFInd = sparkSession.readStream.format("kafka").option("kafka.bootstrap.servers", "localhost:9092")
  .option("subscribe", "IndiaStocks")
  .option("startingOffsets", "earliest")
  .load()

//readStreamDFInd.printSchema()
val readStreamDFUS = sparkSession.readStream.format("kafka").option("kafka.bootstrap.servers", "localhost:9092")
  .option("subscribe", "USStocks")
  .option("startingOffsets", "earliest")
  .load()

val schema = new StructType()
  .add("stock", StringType)
  .add("buy", IntegerType)
  .add("sell", IntegerType)
  .add("profit", IntegerType)
  .add("quantity", IntegerType)
  .add("loss", IntegerType)
  .add("gender", StringType)

val stocksIndia = readStreamDFInd.selectExpr("CAST(value as STRING) as json").select(from_json($"json", schema).as("data")).select("data.*")
val stocksUSA = readStreamDFUS.selectExpr("CAST(value as STRING) as json").select(from_json($"json", schema).as("data")).select("data.*")
stocksIndia.printSchema() stocksUSA.writeStream
  .format("console")
  .outputMode("append").trigger(Trigger.ProcessingTime("5 seconds"))
  .start()
  .awaitTermination()

【问题讨论】：

标签： scala apache-spark apache-kafka apache-spark-sql spark-structured-streaming

【解决方案1】：

代码运行良好，您也可以在 book 中看到。

查看from_json 函数的文档，会创建null 值，因为字符串不可解析。

=> 您在 json 字符串中缺少 quantity 字段周围的引号。

【讨论】：

【解决方案2】：

问题出在您的 kafka 数据中，数量列应该用引号引起来。请在下方查看。

{"stock":"SEE","buy":12,"sell":15,"profit":3,"quantity":27,"loss":0,"性别":"M"} {"stock":"SEE","buy":12,"sell":15,"profit":3,"quantity":27,"loss":0,"gender": “M”} {"stock":"SEE","buy":12,"sell":15,"profit":3,"quantity":27,"loss":0,"gender": “M”}

【讨论】：