链接到 sql 表的增量表答案

【问题标题】：Delta table linked to sql table链接到 sql 表的增量表
【发布时间】：2023-02-10 08:34:20
【问题描述】：

I'm new on databricks and spark, we create delta table using data from sql. Theese table are kind of mirrored. Basicalli if I insert a new row to sql it affects delta, I can even insert from databricks having sql updated, but deleting is allowed only from sql. By the way, I don't understand how it works, if I create a delta table with this command the delta and sql table are linked

spark.sql("""
 create table IF NOT EXISTS dbname.delta_table
 using org.apache.spark.sql.jdbc
 OPTIONS (
   url '""" + sql_url + """',
   dbtable 'dbname.sql_table',
   user '""" + sql_user + """',
   password '""" + sql_password + """',
   TRUNCATE true
 )
""");

But if I try with pyspark, there's no link between table

spark.read \
  .format("jdbc") \
  .option("url", url_sql) \
  .option("dbtable", sql_table) \
  .option("user", sql_user) \
  .option("password", sql_password) \
  .option("truncate", True) \
  .load() \
  .write \
  .saveAsTable(delta_table)

I would like to know how to get the same result with pyspark and how to get more documentation about it, I didn't find what I was looking for, I don't know what kind of relationship there's between table and the keyword related to this. Thanks for help Sergio

I've been looking online all day to find the correct topic but I didn't find anything

It seems that your first SQL Statement is just trying to create a table in the SQL Server. It doesn't create a delta table. Does the SQL Statement run successfully and actually create a table?
Your second command reads the same table from the SQL Server and tries to create a delta table with the name defined by delta_table. The SQL and PySpark scripts are doing two different things. The SQL one is creating a table in SQL Server (with no schema?) and the pyspark one is reading the same table from SQL Server and saving it as a delta table
Actually the SQL script creates a delta table on databricks using the sql table
Are you sure? In this documentation it specifies that if you want to create a delta table, you should either say USING delta or nothing at all. Can you point to the documentation where it tells you that you can create a delta table with that SQL command?

标签： sql-server apache-spark pyspark azure-databricks mssql-jdbc

【解决方案1】：

你在做不同的事情：

第一个 SQL 语句在指向 SQL 数据库的 Hive Metastore 中创建一个元数据条目。因此，当您读取它时，Spark 在后台通过 JDBC 协议连接并加载数据。
在第二种方法中，您实际上是从数据库加载数据，并创建一个以 Delta 格式（默认格式）存储的托管表。这张表是 SQL Server 在执行时的快照。

真的，如果你想像第一种情况一样创建一个表，你只需要继续使用spark.sql。

【讨论】：