从 pyspark 中的数据框数组类型列中获取“名称”元素

Question

我已经生成了 pyspark.sql.dataframe.DataFrame，其中列名称为 cast 和 score。

但是，我想在 _score 列旁边保留 cast 列中的唯一名称，而不是与它们关联的 id。

e.g Liam Neeson, 'Dan Stevens, Marina Squerciati, Scott Frank

cast 列中数据类型的结构如下所示

df_test.schema['cast'].dataType

StructType(List(StructField(Acteur$divActrice,ArrayType(StructType(List(StructField(id,StringType,true),StructField(name,StringType,true))),true),true),StructField(Regisseur,ArrayType(StructType(List(StructField(id,StringType,true),StructField(name,StringType,true))),true),true),StructField(writer,ArrayType(StructType(List(StructField(id,StringType,true),StructField(name,StringType,true))),true),true)))

    elasticsearch_performance_evaluation.drop('id', '_type', 'name').show(20, False)

+------------------------------------------------------------------------------------------------------------+---------+
|cast                                                                                                        |_score   |
+------------------------------------------------------------------------------------------------------------+---------+
|[[[3713, Liam Neeson], [24315, Dan Stevens], [37138, Marina Squerciati]], [[37136, Scott Frank]],]          |90.68749 |
|[[[365, Susan Sarandon], [1939, Tom Hanks], [13425, Halle Berry]], [[6813, Tom Tykwer]],]                   |42.601303|
|[[[1939, Tom Hanks], [3339, Julia Roberts], [8479, Sarah Mahoney]], [[1939, Tom Hanks]],]                   |42.601196|
|[[[1939, Tom Hanks], [42930, Sarita Choudhury], [44288, Alexander Black]], [[6813, Tom Tykwer]],]           |42.601196|
|[[[13379, Emilio Estevez], [13381, Martin Sheen], [13383, Deborah Kara Unger]], [[13379, Emilio Estevez]],] |22.130056|
|[[[15871, Frank Lammers]], [[30157, Mark Mertens]],]                                                        |20.221449|
|[[[44061, Erin Darke], [44548, Hansel Tan], [233, Rachel Weisz]], [[10329, Joshua Marston]],]               |19.26571 |
|[[[1535, Pim Wessels], [2327, Joosje Duk], [2329, Ydwer Bosma]], [[1537, Steven de Jong]],]                 |18.798985|
|[[[6741, Chantal Janzen], [16357, Benja Bruijning], [15865, Fedja van Huêt]], [[2769, Kees van Nieuwkerk]],]|18.79888 |
|[[[6105, Stephen Dorff], [9611, Tom Berenger], [12397, Chyler Leigh]], [[12395, Gabe Torres]],]             |17.88121 |
|[[[2309, Gene Hackman], [8955, Tom Cruise], [12971, Ed Harris]], [[30047, Sydney Pollack]],]                |17.88121 |
|[[[1105, Marisa Tomei], [12763, Hugh Grant], [43842, J.K Simmons]], [[5637, Marc Lawrence]],]               |17.88121 |
|[[[8955, Tom Cruise], [10405, Kelly McGillis], [2023, Val Kilmer]], [[2311, Tony Scott]],]                  |17.88121 |
|[[[2105, Tom Hughes], [43507, Emma Greenwell], [43508, Alex Macqueen]], [[43506, Vivienne de Courcy]],]     |17.881105|
|[[[1201, Carice van Houten], [2271, Bill Nighy], [8955, Tom Cruise]], [[18203, Bryan Singer]],]             |17.881105|
|[[[9159, Anne Hathaway], [9161, Jim Sturgess], [9163, Tom Mison]], [[9157, Lone Scherfig]],]                |17.881105|
|[[[921, Ewan McGregor], [4603, Naomi Watts], [16997, Tom Holland]], [[16995, Juan Antonio Bayona]],]        |17.881105|
|[[[44161, Edie Falco], [44162, Miles Gaston Villanueva], [44163, Gus Halper]],,]                            |17.65994 |
|[[[2327, Joosje Duk], [2329, Ydwer Bosma], [3461, Leo de Jong]], [[1537, Steven de Jong]],]                 |17.65994 |
|[[[10841, Bas Muijs], [25285, Vivian van Huiden], [28091, Tom van Kalmthout]], [[1537, Steven de Jong]],]   |16.626995|
+------------------------------------------------------------------------------------------------------------+---------+

有人可以帮助我如何仅使用演员名称和分数列提取数据。

提前致谢

Answer 1

一种可能的解决方案是转换为 rdd，然后是提取演员名称的映射函数，最后转换回数据帧。

df1 = spark.createDataFrame([([[["3713", "Liam Neeson"], ["24315", "Dan Stevens"],\
 ["37138", "Marina Squerciati"]], [["37136", "Scott Frank"]],], 90.687),\
([[["365", "Susan Sarandon"], ["1939", "Tom Hanks"], ["13425", "Halle Berry"]],\
 [["6813", "Tom Tykwer"]],], 42.601303)],["cast","score"])
#creation of the dataframe, e.g. first 2 rows of your example

def extract_actor(row):                    #extract function
    actors, score = row
    new_actors = list()
    for l in actors:
        res = list()
        for actor in l:
            res.append(actor[1])          #walk through all the lists and collect the names
        new_actors.append(res)
    return (new_actors,score)

df1.rdd.map(extract_actor).toDF(["cast","score"]).show(truncate=False)
#transform dataframe to rdd apply the map and transform result back to dataframe

输出：

+--------------------------------------------------------------------------------------+---------+
|cast                                                                                  |score    |
+--------------------------------------------------------------------------------------+---------+
|[WrappedArray(Liam Neeson, Dan Stevens, Marina Squerciati), WrappedArray(Scott Frank)]|90.687   |
|[WrappedArray(Susan Sarandon, Tom Hanks, Halle Berry), WrappedArray(Tom Tykwer)]      |42.601303|
+--------------------------------------------------------------------------------------+---------+

Answer 2

你可以使用explode函数，更多信息见these答案