Spark + BigQuery：`InvocationTargetException：java.lang.VerifyError：操作数堆栈上的类型错误`

Question

我已经尝试过以下所有组合：

• Python 和 Scala
• JDK 8 和 JDK 11
• Spark 3.1.2 和 Spark 3.3.1

我得到了同样的错误。在这里我将展示 Scala + JDK 11 + Spark 3.3.1 尝试，但正如我所说，所有组合都会导致相同的错误：

1. 为 JDK 11 设置 JAVA_HOME，为 Spark 3.3.1 设置 SPARK_HOME，并运行配置了 BigQuery + GCS 连接器的 Scala spark-shell：

   export JAVA_HOME=$(/usr/libexec/java_home -v 11)
   export SPARK_HOME=~/opt/spark/spark-3.3.1-bin-hadoop3-scala2.13
   $SPARK_HOME/bin/spark-shell \
     -c spark.hadoop.fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS \
     -c spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem \
     --packages "com.google.cloud.spark:spark-bigquery-with-dependencies_2.13:0.28.0,com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.10"
   

   2. 设置虚拟测试数据帧：

   import org.apache.spark.sql._
   import org.apache.spark.sql.types._
   
   val df = spark.createDataFrame(
     java.util.List.of(
       Row(1, "foo"),
       Row(2, "bar")
     ), StructType(
       StructField("a", IntegerType) ::
       StructField("b", StringType) ::
       Nil))
   
   df.show()
   

   结果是：

   +---+---+
   |  a|  b|
   +---+---+
   |  1|foo|
   |  2|bar|
   +---+---+
   

   3. 将简单数据框写入 BigQuery：

   df.write.
     format("bigquery").
     mode("overwrite").
     option("project", "<redacted>").
     option("parentProject", "<redacted>").
     option("dataset", "<redacted>").
     option("credentials", bigquery_credentials_b64).
     option("temporaryGcsBucket", "<redacted>").
     save("test_table")
   

   我得到：

   java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3467)
     at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
     at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
     at com.google.cloud.spark.bigquery.SparkBigQueryUtil.getUniqueGcsPath(SparkBigQueryUtil.java:127)
     at com.google.cloud.spark.bigquery.SparkBigQueryUtil.createGcsPath(SparkBigQueryUtil.java:108)
     ... 75 elided
   Caused by: java.lang.reflect.InvocationTargetException: java.lang.VerifyError: Bad type on operand stack
   Exception Details:
     Location:
       com/google/api/ClientProto.registerAllExtensions(Lcom/google/protobuf/ExtensionRegistryLite;)V @4: invokevirtual
     Reason:
       Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
     Current Frame:
       bci: @4
       flags: { }
       locals: { 'com/google/protobuf/ExtensionRegistryLite' }
       stack: { 'com/google/protobuf/ExtensionRegistryLite', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
     Bytecode:
       0000000: 2ab2 0002 b600 032a b200 04b6 0003 2ab2
       0000010: 0005 b600 03b1
   
     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:135)
     ... 83 more
   Caused by: java.lang.VerifyError: Bad type on operand stack
   Exception Details:
     Location:
       com/google/api/ClientProto.registerAllExtensions(Lcom/google/protobuf/ExtensionRegistryLite;)V @4: invokevirtual
     Reason:
       Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
     Current Frame:
       bci: @4
       flags: { }
       locals: { 'com/google/protobuf/ExtensionRegistryLite' }
       stack: { 'com/google/protobuf/ExtensionRegistryLite', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
     Bytecode:
       0000000: 2ab2 0002 b600 032a b200 04b6 0003 2ab2
       0000010: 0005 b600 03b1
   
     ... 5 elided and 88 more
   

Answer 1

解决方案是需要自定义阴影 .jars。 Databricks 和 Amazon EMR 等托管 Spark 环境已经解决了这些问题，但要在本地环境中使用 spark-shell 运行这实际上是相当复杂的。