'YYYY-MM-dd'和'yyyy-MM-dd'日期格式之间的Spark-sql区别[重复]

Question

我有一个用例向时间戳列添加一些值并从中派生日期。我正在尝试使用 from_unix 和 unix_timestamp 函数。但它以不同的格式显示不同的值。

select from_unixtime('1577582583',  'YYYY-MM-dd'), from_unixtime('1577582583',  'yyyy-MM-dd');
from_unixtime(CAST(1577582583 AS BIGINT), YYYY-MM-dd)   from_unixtime(CAST(1577582583 AS BIGINT), yyyy-MM-dd)
2020-12-29  2019-12-29

有人能区分“YYYY-MM-dd”格式和“yyyy-MM-dd”格式吗？

Answer 1

YYYY 是星期日期，yyyy 是普通日期。你可以在那里检查它： https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html https://en.wikipedia.org/wiki/ISO_week_date

Answer 2

在这种情况下没有区别....但正常的方式是 yyyy 不是 YYYY see the code FromUnixTimestamp here

两种格式的值相同。我认为即使您传递了错误的格式，内部催化剂也会小心...

查看此示例以演示行为：

package com.examples

import org.apache.log4j.{Level, Logger}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

object UnixTimestamp extends App {

  val spark: SparkSession = SparkSession.builder()
    .master("local")
    .appName("UnixTimestamp")
    .getOrCreate()
  Logger.getLogger("org").setLevel(Level.WARN)

  spark.sparkContext.setLogLevel("ERROR")

  import spark.sqlContext.implicits._



  //Convert unix timestamp to date
  val dateDF = Seq(1).toDF("seq").select(
    from_unixtime(unix_timestamp(), "MM-dd-yyyy").as("date_1"),
    from_unixtime(unix_timestamp(), "dd-MM-yyyy HH:mm:ss").as("date_2"),
    from_unixtime(unix_timestamp(), "yyyy-MM-dd").as("date_3"),
    from_unixtime(unix_timestamp(), "YYYY-MM-dd").as("date_4"),
    from_unixtime(lit(1577582583), "YYYY-MM-dd").as("yourdate1"), // your example date format 1
    from_unixtime(lit(1577582583), "yyyy-MM-dd").as("yourdate2")//// your example date format 2

  )
  dateDF.printSchema()
  dateDF.show(false)
  println("lets consider sql or hive way here of what you have done")
  spark.sql("select from_unixtime('1577582583',  'YYYY-MM-dd'), from_unixtime('1577582583',  'yyyy-MM-dd')").show
}

结果：这里 yourdate1 (YYYY-MM-dd) 和 yourdate2 (yyyy-MM-dd) 是两种不同的格式

root
|-- date_1: string (nullable = true)
|-- date_2: string (nullable = true)
|-- date_3: string (nullable = true)
|-- date_4: string (nullable = true)
|-- yourdate1: string (nullable = true)
|-- yourdate2: string (nullable = true)

+----------+-------------------+----------+----------+----------+----------+
|date_1    |date_2             |date_3    |date_4    |yourdate1 |yourdate2 |
+----------+-------------------+----------+----------+----------+----------+
|02-14-2020|14-02-2020 21:19:53|2020-02-14|2020-02-14|2019-12-28|2019-12-28|
+----------+-------------------+----------+----------+----------+----------+

lets consider sql or hive way here of what you have done
+-----------------------------------------------------+-----------------------------------------------------+
|from_unixtime(CAST(1577582583 AS BIGINT), YYYY-MM-dd)|from_unixtime(CAST(1577582583 AS BIGINT), yyyy-MM-dd)|
+-----------------------------------------------------+-----------------------------------------------------+
|                                           2019-12-28|                                           2019-12-28|
+-----------------------------------------------------+-----------------------------------------------------+