使用 Azure SQL DB MetaException 的 Databricks 外部配置单元元存储（消息：在元存储中找不到版本信息。）

Question

我正在尝试使用以下初始化脚本为 Databricks 设置外部配置单元元存储：

dbutils.fs.put(
"/databricks/scripts/external-metastore.sh",
"""#!/bin/sh
  |# Loads environment variables to determine the correct JDBC driver to use.
  |source /etc/environment
  |# Quoting the label (i.e. EOF) with single quotes to disable variable interpolation.
  |cat << 'EOF' > /databricks/driver/conf/00-custom-spark.conf
  |[driver] {
  |    # Hive specific configuration options.
  |    # spark.hadoop prefix is added to make sure these Hive specific options will propagate to the metastore client.
  |    # JDBC connect string for a JDBC metastore
  |    "spark.hadoop.javax.jdo.option.ConnectionURL" = "jdbc:sqlserver://tst.database.windows.net:1433;database=db;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net"
  |
  |    # Username to use against metastore database
  |    "spark.hadoop.javax.jdo.option.ConnectionUserName" = "admin"
  |
  |    # Password to use against metastore database
  |    "spark.hadoop.javax.jdo.option.ConnectionPassword" = "password"
  |
  |    # Driver class name for a JDBC metastore
  |    "spark.hadoop.javax.jdo.option.ConnectionDriverName" = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
  |
  |    # Spark specific configuration options
  |    "spark.sql.hive.metastore.version" = "2.3.7"
  |    # Skip this one if <hive-version> is 0.13.x.
  |    "spark.sql.hive.metastore.jars" = "builtin"
  |}
  |EOF
  |""".stripMargin,
overwrite = true)

我运行这个 Scala 代码

spark.conf.get("spark.sql.hive.metastore.version")

哪个输出：

res0: String = 2.3.7

但是当我尝试运行时：

spark.table("diamonds").withColumnRenamed("table", "table_number")
 .write
 .jdbc(jdbcUrl, "diamonds", connectionProperties)

我收到此错误：

AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
Caused by: HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: InvocationTargetException: 
Caused by: MetaException: Version information not found in metastore. 

Answer 1

仅来自文档External Apache Hive metastore：

• SQL Server 不能用作底层元存储数据库 Hive 2.0 及更高版本。

请尝试更改版本。在这里您可以参考另一个相同的问题： Azure Databricks external Hive Metastore

HTH。