无法从 azure blob 存储容器读取 xlsx 文件到 pyspark 数据帧

Question

我正在尝试将数据从 Azure 存储容器加载到 Azure Databricks 中的 Pyspark 数据框。当我读取 txt 或 CSV 文件时，它正在工作。但是当我尝试读取 .xlsx 文件时，我遇到了以下问题。

阿帕奇星火 3.2.0，斯卡拉 2.12

以下是我正在执行的步骤

spark.conf.set("fs.azure.account.key.teststorage.blob.core.windows.net",
"**********************")

这是工作

df = spark.read.format("csv").option("header", "true") \
  .option("inferSchema", "true") \
  .load("wasbs://testcontainer@teststorage.blob.core.windows.net/data/samplefile.txt")

不工作

df = spark.read.format("com.crealytics.spark.excel") \
  .option("header", "true").option("inferSchema","true") \
  .load("wasbs://testcontainer@teststorage.blob.core.windows.net/data/samplefile.xlsx")

加载 xlsx 文件时出现以下错误：

: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Container producer in account teststorage.blob.core.windows.net not found, and we can't create it using anoynomous credentials, and no credentials found for them in the configuration.
    at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1063)
    at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:512)
at shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem.initialize(NativeAzureFileSystem.java:1384)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:537)
at com.crealytics.spark.excel.WorkbookReader$.readFromHadoop$1(WorkbookReader.scala:35)
at com.crealytics.spark.excel.WorkbookReader$.$anonfun$apply$2(WorkbookReader.scala:41)
at com.crealytics.spark.excel.DefaultWorkbookReader.$anonfun$openWorkbook$1(WorkbookReader.scala:49)
at scala.Option.fold(Option.scala:251)
at com.crealytics.spark.excel.DefaultWorkbookReader.openWorkbook(WorkbookReader.scala:49)
at com.crealytics.spark.excel.WorkbookReader.withWorkbook(WorkbookReader.scala:14)
at com.crealytics.spark.excel.WorkbookReader.withWorkbook$(WorkbookReader.scala:13)
at com.crealytics.spark.excel.DefaultWorkbookReader.withWorkbook(WorkbookReader.scala:45)
at com.crealytics.spark.excel.ExcelRelation.excerpt$lzycompute(ExcelRelation.scala:31)
at com.crealytics.spark.excel.ExcelRelation.excerpt(ExcelRelation.scala:31)
at com.crealytics.spark.excel.ExcelRelation.headerColumns$lzycompute(ExcelRelation.scala:102)
at com.crealytics.spark.excel.ExcelRelation.headerColumns(ExcelRelation.scala:101)
at com.crealytics.spark.excel.ExcelRelation.$anonfun$inferSchema$1(ExcelRelation.scala:163)
at scala.Option.getOrElse(Option.scala:189)
at com.crealytics.spark.excel.ExcelRelation.inferSchema(ExcelRelation.scala:162)
at com.crealytics.spark.excel.ExcelRelation.<init>(ExcelRelation.scala:35)
at com.crealytics.spark.excel.DefaultSource.createRelation(DefaultSource.scala:35)
at com.crealytics.spark.excel.DefaultSource.createRelation(DefaultSource.scala:13)
at com.crealytics.spark.excel.DefaultSource.createRelation(DefaultSource.scala:8)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:385)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:355)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:322)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:322)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:235)
at sun.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
at py4j.Gateway.invoke(Gateway.java:295)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)

笔记：我能够从 dbfs 和挂载点读取。

Answer 1

出现此问题是由于 blob 存储容器的公共访问级别。当容器具有Private或Blob公共访问级别时，excel 文件会出现相同的错误。但是当使用Container公共访问级别的blob存储容器时，您将能够无误地读取excel文件。这是我在尝试重现问题时得到的。

一个简单的解决方案是将容器的公共访问级别更改为Container，或者将 blob 存储帐户挂载到 Databricks 文件系统（适合您）。如果您选择更改容器的公共访问级别，请转到您的 blob 存储中的容器，您将找到“更改访问级别”选项，您可以在其中选择 Container 级别。

• 导航到存储帐户的容器并更改访问级别。

• 返回 Databricks，再次运行 Dataframe read，它没有任何错误。

df2 = spark.read.format("com.crealytics.spark.excel") 
      .option("header", "true").option("inferSchema","true") 
      .load("wasbs://<container>@<storage_acc>.blob.core.windows.net/data.xlsx")

请参阅以下文档以了解有关使用 Databricks 访问 blob 存储帐户的更多信息。

https://learn.microsoft.com/en-us/azure/databricks/data/data-sources/azure/azure-storage