Scala - 使用表达式的评估将数据帧写入 csv 文件

Question

这是使用表达式（字符串）的评估（Eval 或类似的东西）将数据帧写入 Scala 中的 csv 文件。

 import org.apache.spark.sql.{SaveMode, SparkSession, SQLContext, Row, DataFrame, Column}
 import scala.reflect.runtime.universe._
 import scala.tools.reflect.ToolBox
 import scala.reflect.runtime.currentMirror

 val df = Seq(("a", "b", "c"), ("a1", "b1", "c1")).toDF("A", "B", "C")
 val df_write = """df.coalesce(1).write.option("delimiter", "\u001F").csv("file:///var/tmp/test")"""

 // This is one of my failed attempts - I have tried using the interpreter as well (code not shown here).    
 val tb = runtimeMirror(getClass.getClassLoader).mkToolBox()  
 toolbox.eval(toolbox.parse(df_write))

 Errors are: 
 object coalesce is not a member of package df ....

Answer 1

Shiva，试试下面的代码。问题是对象变量不在工具箱的范围内，因此无法计算表达式。

package com.mansoor.test

import org.apache.spark.sql.{DataFrame, SparkSession}

object Driver extends App {

  def evalCode[T](code: String): T = {
    import scala.tools.reflect.ToolBox
    import scala.reflect.runtime.{currentMirror => m}
    val toolbox = m.mkToolBox()
    toolbox.eval(toolbox.parse(code)).asInstanceOf[T]
  }

  val sparkSession: SparkSession = SparkSession.builder().appName("Test")
    .master("local[2]")
    .getOrCreate()

  import sparkSession.implicits._
  val df: DataFrame = Seq(("a", "b", "c"), ("a1", "b1", "c1")).toDF("A", "B", "C")

  val df_write =
    s"""
       |import com.mansoor.test.Driver._
       |
       |df.coalesce(1).write.option("delimiter", "\u001F").csv("file:///var/tmp/test")
       """.stripMargin

  evalCode[Unit](df_write)

  sparkSession.sparkContext.stop()
}