org.apache.spark.sql.AnalysisException：流数据帧/数据集不支持非基于时间的窗口；；尽管有基于时间的窗口答案

【问题标题】：org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;; despite of time-based windoworg.apache.spark.sql.AnalysisException：流数据帧/数据集不支持非基于时间的窗口；；尽管有基于时间的窗口
【发布时间】：2026-02-03 09:50:01
【问题描述】：

我正在为 Spark Structured Streaming 进行基于窗口的排序：

val filterWindow: WindowSpec = Window
  .partitionBy("key")
  .orderBy($"time")

controlDataFrame=controlDataFrame.withColumn("Make Coffee", $"value").    
  withColumn("datetime", date_trunc("second", current_timestamp())).
  withColumn("time", current_timestamp()).
  withColumn("temp_rank", rank().over(filterWindow))
  .filter(col("temp_rank") === 1)
  .drop("temp_rank").
  withColumn("digitalTwinId", lit(digitalTwinId)).
  withWatermark("datetime", "10 seconds")

我将time 获取为current_timestamp()，在架构中我看到它的类型为StructField(time,TimestampType,true)

为什么 Spark 3.0 不允许我基于它进行窗口操作，除了以下异常，因为该字段显然是基于时间的？

21/11/22 10:34:03 WARN SparkSession$Builder: Using an existing SparkSession; some spark core configurations may not take effect.

org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;;
Window [rank(time#163) windowspecdefinition(key#150, time#163 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS temp_rank#171], [key#150], [time#163 ASC NULLS FIRST]
+- Project [key#150, value#151, Make Coffee#154, datetime#158, time#163]

【问题讨论】：

标签： scala apache-spark spark-streaming

【解决方案1】：

所以，根据https://issues.apache.org/jira/browse/SPARK-37439?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel的回答

基于时间的窗口不仅仅是基于时间戳格式列的窗口/窗口规范。您应该为时间戳列显式使用session_window() 函数。

http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#types-of-time-windows

【讨论】：