在 pyspark 中使用 f.coalesce 后如何获取值的原始数据类型？

Question

list = ["B", "A", "D", "C"]

data = [("B", "On","NULL",1632733508,"active"),
        ("B", "Off","NULL",1632733508, "active"), 
        ("A","On","NULL",1632733511,"active"),    
        ("A","Off","NULL",1632733512,"active"),
        ("D","NULL",450,1632733513,"inactive"),
        ("D","NULL",431,1632733515,"inactive"),
        ("C","NULL",20,1632733518,"inactive"),
        ("C","NULL",30,1632733521,"inactive")]

df = spark.createDataFrame(data, ["unique_string", "ID", "string_value", "numeric_value", "timestamp","mode"])

为了根据列表拆分 df，我有以下代码。

    split_df = (df.filter(
                f.col('listname') == list)
                .select(
                f.coalesce(f.col('string_value'), 
                f.col('double_value')).alias(list), 
                f.col('timestamp'), f.col('mode')
               ))
               return split_df


dfs = [split_df(df, list) for id in list]

起点

    ID  string_value    numeric_value   timestamp   mode   
0   B   On              NULL            1632733508  active
1   B   Off             NULL            1632733508  active
2   A   On              NULL            1632733511  active
3   A   Off             NULL            1632733512  active
4   D   NULL            450             1632733513  inactive
5   D   NULL            431             1632733515  inactive
6   C   NULL            20              1632733518  inactive
7   C   NULL            30              1632733521  inactive

使用函数 split_df 后，有一个 df 列表，如下所示。

dfs[1].show()

    D   timestamp   mode
0   450 1632733513  inactive
1   431 1632733515  inactive

使用 f.coalesce 后，每列中的所有值都将是一个字符串。对于像 ID“D”这样的数字变量，这并不好。正如 printSchema 所示，ID "D" 是一个字符串而不是双精度字符串，而 "timestamp" 也是一个字符串而不是长字符串。

dfs[1].printSchema()

root
 |-- D: string (nullable = true)
 |-- timestamp: string (nullable = true)
 |-- mode: string (nullable = true)

保留原始数据类型的函数有什么用？

Answer 1

您可以将其cast 设置为您想要的任何数据类型，例如

f.coalesce(
  f.col('str_col'),
  f.col('int_col'),
).cast('int')