Pyspark 过滤器性能 3.0.1 与 3.1.2答案

【问题标题】：Pyspark filter performance 3.0.1 vs 3.1.2Pyspark 过滤器性能 3.0.1 与 3.1.2
【发布时间】：2021-09-17 14:04:35
【问题描述】：

我在本地运行我的应用程序。我有一个包含 3 列的数据框

df.show()
+--+---------+--------------+
|ID|      key|           val|
+--+---------+--------------+
|12|COL1_FLAG|         Valid|
|12|COL2_FLAG|         Valid|
|12|COL3_FLAG|Invalid Format|
+--+---------+--------------+

df_filtered = df.filter(col('val').like('Invalid%'))

当我的 spark 版本指向 3.0.1 时，上述过滤器工作正常。当我将本地 spark 安装指向 3.1.2 时，执行在过滤阶段挂起。我什至尝试将过滤器更改为

df_filtered = df.filter(col('val').substr(1,7) == 'Invalid')

行为没有变化。我在这里做错了什么还是有更好的方法来实现这个过滤器？

3.1.2 上的 explain() 失败并出现以下错误

py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.sql.api.python.PythonSQLUtils.explainString.
: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at org.apache.spark.sql.catalyst.trees.TreeNode$$Lambda$949/1910987899.get$Lambda(Unknown Source)
        at java.lang.invoke.LambdaForm$DMH/2088051243.invokeStatic_LL_L(LambdaForm$DMH)
        at java.lang.invoke.LambdaForm$MH/112061925.linkToTargetMethod(LambdaForm$MH)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:318)
        at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$3(TreeNode.scala:323)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$Lambda$951/1279057069.apply(Unknown Source)
        at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:408)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$Lambda$1229/1925318585.apply(Unknown Source)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:244)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:406)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:359)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:323)
        at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$3(TreeNode.scala:323)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$Lambda$951/1279057069.apply(Unknown Source)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChild$2(TreeNode.scala:386)
        at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$4(TreeNode.scala:438)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$Lambda$1238/1680937321.apply(Unknown Source)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
        at scala.collection.TraversableLike$$Lambda$26/632587706.apply(Unknown Source)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at scala.collection.TraversableLike.map(TraversableLike.scala:238)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:438)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$Lambda$1229/1925318585.apply(Unknown Source)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:244)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:406)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:359)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:323)
        at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$3(TreeNode.scala:323)

【问题讨论】：

对每个版本运行此命令df_filtered.explain() 并更新您的问题
用 3.1.2 的解释计划更新了我的问题
OutOfMemory 执行explain 方法，我从没见过这样的

标签： pyspark

【解决方案1】：

我们遇到了同样的问题。将spark.sql.constraintPropagation.enabled 设置为false

【讨论】：