spark sql中的BIGINT和INT比较失败答案

【问题标题】：BIGINT and INT comparison failure in spark sqlspark sql中的BIGINT和INT比较失败
【发布时间】：2016-08-17 14:23:27
【问题描述】：

我有一个具有以下定义的 Hive 表：

create table testforerror (
    my_column MAP<BIGINT, ARRAY<String>>
);

该表有以下记录

hive> select * from testforerror;
OK
{16001:["0034000000a4WDAAA2"]}
{16001:["0034000000orWiFAAU"]}
{16001:["","0034000000VgrHdAAJ"]}
{16001:["0034000000cS4tDAAS"]}
{15001:["0037000001a7ofgAAA"]}
Time taken: 0.067 seconds, Fetched: 5 row(s)

我有一个查询，它使用 my_column 的键过滤记录。查询如下

select * from testforerror where my_column[16001] is not null;

此查询在 hive/beeline shell 中执行良好并产生以下记录：

hive> select * from testforerror where my_column[16001] is not null;
OK
{16001:["0034000000a4WDAAA2"]}
{16001:["0034000000orWiFAAU"]}
{16001:["","0034000000VgrHdAAJ"]}
{16001:["0034000000cS4tDAAS"]}
Time taken: 2.224 seconds, Fetched: 4 row(s)

但是，当我尝试从 spark sqlContext 执行时出现错误。以下是错误信息：

scala> val errorquery = "select * from testforerror where my_column[16001] is not null"
errorquery: String = select * from testforerror where my_column[16001] is not null

scala> sqlContext.sql(errorquery).show()
org.apache.spark.sql.AnalysisException: cannot resolve 'my_column[16001]' due to data type mismatch: argument 2 requires bigint type, however, '16001' is of int type.; line 1 pos 43
    at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
    at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:65)
    at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:57)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:335)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:335)
    at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:334)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:332)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:332)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:281)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)

任何指针都非常有帮助，谢谢。

【问题讨论】：

标签： apache-spark hive apache-spark-sql

【解决方案1】：

您可以使用 DSL 和 getItem 方法代替 SQL 和括号表示法：

sqlContext.table("testforerror").where($"mycolumn".getItem(1L).isNotNull)

【讨论】：

这会有所帮助，但我无法切换到 DSL，因为我的查询非常复杂，我试图掩盖细节。谢谢。
我认为目前没有其他选择。不过，您应该为此打开一个JIRA。