无法使用外部配置单元元存储配置数据块

Question

我正在关注这份文件https://docs.databricks.com/data/metastores/external-hive-metastore.html#spark-configuration-options 连接到我的外部配置单元元存储。我的 Metastore 版本是 3.1.0 并遵循文档。

docs.databricks.comdocs.databricks.com 外部 Apache Hive 元存储 — Databricks 文档 了解如何连接到 Databricks 中的外部 Apache Hive 元存储。 10:51 尝试连接到外部配置单元元存储时出现此错误

org/apache/hadoop/hive/conf/HiveConf when creating Hive client using classpath: 
Please make sure that jars for your version of hive and hadoop are included in the paths passed to spark.sql.hive.metastore.jars

spark.sql.hive.metastore.jars=/databricks/hive_metastore_jars/*

当我在 /databricks/hive_metastore_jars/ 上执行 ls 时，我可以看到所有复制的文件 10:52 我是否需要复制任何 Hive 特定文件并将其上传到此文件夹中？

我完全按照网站上的说明做了

这些是我的 hive_metastore_jars 的内容

total 56K
drwxr-xr-x 3 root root 4.0K Mar 24 05:06 1585025573715-0
drwxr-xr-x 2 root root 4.0K Mar 24 05:06 d596a6ec-e105-4a6e-af95-df3feffc263d_resources
drwxr-xr-x 3 root root 4.0K Mar 24 05:06 repl
drwxr-xr-x 2 root root 4.0K Mar 24 05:06 spark-2959157d-2018-441a-a7d3-d7cecb8a645f
drwxr-xr-x 4 root root 4.0K Mar 24 05:06 root
drwxr-xr-x 2 root root 4.0K Mar 24 05:06 spark-30a72ee5-304c-432b-9c13-0439511fb0cd
drwxr-xr-x 2 root root 4.0K Mar 24 05:06 spark-a19d167b-d571-4e58-a961-d7f6ced3d52f
-rwxr-xr-x 1 root root 5.5K Mar 24 05:06 _CleanRShell.r3763856699176668909resource.r
-rwxr-xr-x 1 root root 9.7K Mar 24 05:06 _dbutils.r9057087446822479911resource.r
-rwxr-xr-x 1 root root  301 Mar 24 05:06 _rServeScript.r1949348184439973964resource.r
-rwxr-xr-x 1 root root 1.5K Mar 24 05:06 _startR.sh5660449951005543051resource.r

我错过了什么吗？

奇怪的是，如果我查看集群启动日志，我会得到以下结果

20/03/24 07:29:05 INFO Persistence: Property spark.hadoop.javax.jdo.option.ConnectionDriverName unknown - will be ignored
20/03/24 07:29:05 INFO Persistence: Property spark.hadoop.javax.jdo.option.ConnectionURL unknown - will be ignored
20/03/24 07:29:05 INFO Persistence: Property spark.hadoop.javax.jdo.option.ConnectionUserName unknown - will be ignored
20/03/24 07:29:05 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
20/03/24 07:29:05 INFO Persistence: Property spark.hadoop.javax.jdo.option.ConnectionPassword unknown - will be ignored
20/03/24 07:29:05 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
20/03/24 07:29:05 INFO Persistence: Property datanucleus.schema.autoCreateAll unknown - will be ignored

20/03/24 07:29:09 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
20/03/24 07:29:09 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException

上面的配置我已经设置好了，日志里也显示了

20/03/24 07:28:59 INFO SparkContext: Spark configuration:
spark.hadoop.javax.jdo.option.ConnectionDriverName=org.mariadb.jdbc.Driver
spark.hadoop.javax.jdo.option.ConnectionPassword=*********(redacted)
spark.hadoop.javax.jdo.option.ConnectionURL=*********(redacted)
spark.hadoop.javax.jdo.option.ConnectionUserName=*********(redacted)

我的 hive 元存储中也提供了版本信息，我可以连接到 mysql 并查看它显示 架构版本：3.1.0 VER_ID = 1

Answer 1

从输出看，jar 似乎没有复制到“/databricks/hive_metastore_jars/”位置。如您共享的文档链接中所述：

1. 将 spark.sql.hive.metastore.jars 设置为 maven
2. 使用上述配置重启集群，然后查看 Spark 驱动程序日志中的消息：

17/11/18 22:41:19 INFO IsolatedClientLoader: Downloaded metastore jars to <path>

从此位置将 jar 从同一集群复制到 DBFS，然后使用初始化脚本将 jar 从 DBFS 复制到“/databricks/hive_metastore_jars/”

Answer 2

当我使用 azure mysql 时，我还需要执行一个步骤 https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore