如何从现有的时间戳列将新的日期时间列添加到火花数据帧

Question

我在 Spark 中有一个具有列时间戳的数据框。我想向这个数据框添加一个新列，该数据框的 DateTime 采用以下格式，从这个现有的时间戳列创建。

“YYYY-MM-DD HH:MM:SS”

Answer 1

您可以使用to_timestamp -

>>> from pyspark import SparkContext
>>> from pyspark.sql import SQLContext
>>> from functools import reduce
>>> import pyspark.sql.functions as F
>>> 
>>> sc = SparkContext.getOrCreate()
>>> sql = SQLContext(sc)
>>> 
>>> 
>>> input_list = [
...   (1,"2019-11-07 10:30:00")
...   ,(1,"2019-11-08 10:30:00")
...   ,(1,"2019-11-09 10:30:00")
...   ,(1,"2019-11-11 10:30:00")
...   ,(1,"2019-11-12 10:30:00")
...   ,(1,"2019-11-13 10:30:00")
...   ,(1,"2019-11-14 10:30:00")
...   ,(2,"2019-11-08 10:30:00")
...   ,(2,"2019-11-09 10:30:00")
...   ,(3,"2019-11-09 10:30:00")
...   ,(3,"2019-11-10 10:30:00")
...   ,(3,"2019-11-11 10:30:00")
...   ,(2,"2019-11-15 10:30:00")
...   ,(2,"2019-11-18 10:30:00")
...   ,(4,"2019-11-10 10:30:00")
...   ,(4,"2019-11-11 10:30:00")
... ]
>>> 
>>> 
>>> sparkDF = sql.createDataFrame(input_list,['customerid','date'])
>>> 
>>> sparkDF = sparkDF.withColumn('date_timestamp',F.to_timestamp(F.col('date'), 'yyyy-MM-dd HH:mm:ss'))
>>> sparkDF.show()
+----------+-------------------+-------------------+
|customerid|               date|     date_timestamp|
+----------+-------------------+-------------------+
|         1|2019-11-07 10:30:00|2019-11-07 10:30:00|
|         1|2019-11-08 10:30:00|2019-11-08 10:30:00|
|         1|2019-11-09 10:30:00|2019-11-09 10:30:00|
|         1|2019-11-11 10:30:00|2019-11-11 10:30:00|
|         1|2019-11-12 10:30:00|2019-11-12 10:30:00|
|         1|2019-11-13 10:30:00|2019-11-13 10:30:00|
|         1|2019-11-14 10:30:00|2019-11-14 10:30:00|
|         2|2019-11-08 10:30:00|2019-11-08 10:30:00|
|         2|2019-11-09 10:30:00|2019-11-09 10:30:00|
|         3|2019-11-09 10:30:00|2019-11-09 10:30:00|
|         3|2019-11-10 10:30:00|2019-11-10 10:30:00|
|         3|2019-11-11 10:30:00|2019-11-11 10:30:00|
|         2|2019-11-15 10:30:00|2019-11-15 10:30:00|
|         2|2019-11-18 10:30:00|2019-11-18 10:30:00|
|         4|2019-11-10 10:30:00|2019-11-10 10:30:00|
|         4|2019-11-11 10:30:00|2019-11-11 10:30:00|
+----------+-------------------+-------------------+

>>>