Spark 2.1 DB2 驱动程序问题答案

【问题标题】：Spark 2.1 DB2 driver IssueSpark 2.1 DB2 驱动程序问题
【发布时间】：2017-07-12 03:02:50
【问题描述】：

在 IBM DSX 上，对于 Python-Spark 2.1 内核，我得到以下代码：

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

# @hidden_cell
# The following code is used to access your data and contains your credentials.
# You might want to remove those credentials before you share your notebook.

properties = {
    'jdbcurl': 'JDBCURL',
    'user': 'USER',
    'password': 'PASSWORD'
}

data_df_1 = spark.read.jdbc(properties['jdbcurl'], table='USER.TABLE', properties=properties)
data_df_1.head()

这个错误：

Py4JJavaError: An error occurred while calling o82.jdbc.
: java.sql.SQLException: No suitable driver
    at java.sql.DriverManager.getDriver(DriverManager.java:326)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$7.apply(JDBCOptions.scala:84)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$7.apply(JDBCOptions.scala:84)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:83)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:34)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:32)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:330)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:125)
    at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:166)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
    at java.lang.reflect.Method.invoke(Method.java:507)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:280)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:214)
    at java.lang.Thread.run(Thread.java:785)

我只在第一次执行时收到此错误。但是，当我再次执行相同的代码时，它就可以工作了。

【问题讨论】：

标签： apache-spark pyspark data-science-experience

【解决方案1】：

这看起来像 https://issues.apache.org/jira/browse/SPARK-19209 。该问题中建议的解决方法是提供“驱动程序”选项。

在您的示例中，如果您需要 DB2 驱动程序：

properties = {
   'jdbcurl': 'JDBCURL',
   'user': 'USER',
   'password': 'PASSWORD',
   'driver': 'com.ibm.db2.jcc.DB2Driver'
}

【讨论】：