org.apache.spark.sql.AnalysisException：

Question

 df.withColumn(x, when($"x" > 75, $"x" + 10).otherwise($"x")).show()
org.apache.spark.sql.AnalysisException: cannot resolve '`x`' given input columns: [Name, Subject, Marks];;
'Project [Name#7, Subject#8, CASE WHEN ('x > 75) THEN ('x + 10) ELSE 'x END AS Marks#38]

scala> df.show()
+----+-------+-----+
|Name|Subject|Marks|
+----+-------+-----+
| Ram|Physics|   80|
|Sham|English|   90|
|Ayan|   Math|   70|
+----+-------+-----+


scala> x
res6: String = Marks

我想传递一个变量作为参数并存储数据框的列值。并根据该参数检查条件、计算值并替换该数据框中具有相同名称的列。

实际上更大的问题是，有多个同名的列，例如 "col1","col2","col3".... 我会将这些列存储在一个数组中，并通过传递数据帧操作中的数组。但就目前而言。如果可以在 spark-scala 中处理，请告诉我问题的解决方案。

Answer 1

试试 String interpolation col({s"${x}"})。

Example:

val df=Seq(("Ram","Physics",80),("Sham","English",90),("Ayan","Math",70)).toDF("Name","Subject","Marks")

df.show()
//+----+-------+-----+
//|Name|Subject|Marks|
//+----+-------+-----+
//| Ram|Physics|   80|
//|Sham|English|   90|
//|Ayan|   Math|   70|
//+----+-------+-----+

import org.apache.spark.sql.functions._
val x:String = "Marks"

df.withColumn(x, when(col(s"${x}") > 75, col(s"${x}") + 10).otherwise(col(s"${x}"))).show()
//+----+-------+-----+
//|Name|Subject|Marks|
//+----+-------+-----+
//| Ram|Physics|   90|
//|Sham|English|  100|
//|Ayan|   Math|   70|
//+----+-------+-----+

Answer 2

为了更好地理解，我将列分隔为 requiredColumns 和 allColumns。

检查下面的代码。

scala> df.show(false)
+----+-------+-----+
|Name|Subject|Marks|
+----+-------+-----+
|Ram |Physics|80   |
|Sham|English|90   |
|Ayan|Math   |70   |
+----+-------+-----+

scala> val requiredColumns = Set("Marks")
requiredColumns: scala.collection.immutable.Set[String] = Set(Marks)

scala> val allColumns = df.columns
allColumns: Array[String] = Array(Name, Subject, Marks)

scala> 
val columnExpr =  allColumns
                    .filterNot(requiredColumn(_))
                    .map(col(_)) ++ requiredColumns
                    .map(c => when(col(c) > 75,col(c) + 10).otherwise(col(c)).as(c))

输出

scala> df.select(columnExpr:_*).show(false)
+----+-------+-----+
|Name|Subject|Marks|
+----+-------+-----+
|Ram |Physics|90   |
|Sham|English|100  |
|Ayan|Math   |70   |
+----+-------+-----+

Answer 3

如下使用functions.col-

 df1.show(false)

    /**
      * +----+-------+-----+
      * |Name|Subject|Marks|
      * +----+-------+-----+
      * |Ram |Physics|80   |
      * |Sham|English|90   |
      * |Ayan|Math   |70   |
      * +----+-------+-----+
*/
val x = "Marks"
    // use functions.col
    df1.withColumn(x, when(col(x) > 75, col(x) + 10).otherwise(col(x)))
      .show()

    /**
      * +----+-------+-----+
      * |Name|Subject|Marks|
      * +----+-------+-----+
      * | Ram|Physics|   90|
      * |Sham|English|  100|
      * |Ayan|   Math|   70|
      * +----+-------+-----+
      */