将日期转换为整数 pyspark

Question

是否可以将日期列转换为 pyspark 数据框中的整数列？我尝试了 2 种不同的方法，但每次尝试都会返回一个包含空值的列。我错过了什么？

from pyspark.sql.types import *

# DUMMY DATA
simpleData = [("James",34,"2006-01-01","true","M",3000.60),
    ("Michael",33,"1980-01-10","true","F",3300.80),
    ("Robert",37,"1992-07-01","false","M",5000.50)
  ]

columns = ["firstname","age","jobStartDate","isGraduated","gender","salary"]
df = spark.createDataFrame(data = simpleData, schema = columns)
df=df.withColumn("jobStartDate", df['jobStartDate'].cast(DateType()))

# ATTEMPT 1 with cast()

df=df.withColumn("jobStartDateAsInteger1", df['jobStartDate'].cast(IntegerType()))

# ATTEMPT 2 with selectExpr()

df=df.selectExpr("*","CAST(jobStartDate as int) as jobStartDateAsInteger2")
df.show()

Answer 1

您可以尝试使用 F.unix_timestamp() 将其转换为 UNIX 时间戳：

from pyspark.sql.types import *
import pyspark.sql.functions as F

# DUMMY DATA
simpleData = [("James",34,"2006-01-01","true","M",3000.60),
    ("Michael",33,"1980-01-10","true","F",3300.80),
    ("Robert",37,"1992-07-01","false","M",5000.50)
  ]

columns = ["firstname","age","jobStartDate","isGraduated","gender","salary"]
df = spark.createDataFrame(data = simpleData, schema = columns)
df=df.withColumn("jobStartDate", df['jobStartDate'].cast(DateType()))

df=df.withColumn("jobStartDateAsInteger1", F.unix_timestamp(df['jobStartDate']))
df.show()

+---------+---+------------+-----------+------+------+----------------------+
|firstname|age|jobStartDate|isGraduated|gender|salary|jobStartDateAsInteger1|
+---------+---+------------+-----------+------+------+----------------------+
|    James| 34|  2006-01-01|       true|     M|3000.6|            1136073600|
|  Michael| 33|  1980-01-10|       true|     F|3300.8|             316310400|
|   Robert| 37|  1992-07-01|      false|     M|5000.5|             709948800|
+---------+---+------------+-----------+------+------+----------------------+