从临时表并发更新插入期间 Postgres 死锁

Question

我有一个由 Airflow 控制的进程，它生成许多任务，执行并发插入 Postgres 数据库。

每个任务都需要一个 pandas 数据框，将行插入到临时表中，然后从临时表中更新插入到目标表中。这会导致死锁，但我很难理解如何缓解这个问题。我已经提取了这里的重要部分，但如果我没有包含足够的信息，请告诉我。

我在 python 3.8.2、postgres 11.7、airflow 1.10.10 中，并使用 psycopg2 作为 odbc 连接。

# create temp table like target table
temp_table_sql = 'CREATE TEMP TABLE mur_global_raw_tmp_61400102 (Like mur_global_raw INCLUDING IDENTITY);'
cur.execute(temp_table_sql)

# serialize dataframe and copy to temp table
pd_df_serial = StringIO()
pd_df.to_csv(pd_df_serial, sep='\t', header=False, index=False)
pd_df_serial.seek(0)

cur.copy_from(pd_df_serial, temp_table_name, null="", columns=pd_df.columns.to_list())
conn.commit()

# upsert from temp table to target table
pd_df_insert_sql = 'INSERT INTO mur_global_raw(lat,lon,time,analysed_sst)
    (SELECT lat,lon,time,analysed_sst FROM mur_global_raw_tmp_61400102
    as tmp_vals ORDER BY lat,lon,time,analysed_sst)
    ON CONFLICT DO NOTHING;'

cur.execute(pd_df_insert_sql)
conn.commit()

这是临时表的架构。

   Column    |           Type           | Collation | Nullable |             Default              | Storage | Stats target | Description
--------------+--------------------------+-----------+----------+----------------------------------+---------+--------------+-------------
 ind          | bigint                   |           | not null | generated by default as identity | plain   |              |
 lat          | double precision         |           |          |                                  | plain   |              |
 lon          | double precision         |           |          |                                  | plain   |              |
 time         | timestamp with time zone |           |          |                                  | plain   |              |
 analysed_sst | double precision         |           |          |                                  | plain   |              |

这是目标表的架构。

    Column    |           Type           | Collation | Nullable |             Default              | Storage | Stats target | Description
--------------+--------------------------+-----------+----------+----------------------------------+---------+--------------+-------------
 ind          | bigint                   |           | not null | generated by default as identity | plain   |              |
 lat          | double precision         |           |          |                                  | plain   |              |
 lon          | double precision         |           |          |                                  | plain   |              |
 time         | timestamp with time zone |           |          |                                  | plain   |              |
 analysed_sst | double precision         |           |          |                                  | plain   |              |
Indexes:
    "mur_global_raw_pkey" PRIMARY KEY, btree (ind)

最后，这是来自服务器日志的示例：

2020-06-22 23:03:36 UTC::@:[3570]:LOG: checkpoint starting: xlog
2020-06-22 23:03:42 UTC:xxxxx(38068):postgres@public_data_raw:[13975]:WARNING: there is no transaction in progress
2020-06-22 23:03:43 UTC:xxxxx(38090):postgres@public_data_raw:[13993]:ERROR: deadlock detected
2020-06-22 23:03:43 UTC:xxxxx(38090):postgres@public_data_raw:[13993]:DETAIL: Process 13993 waits for ShareLock on transaction 42977; blocked by process 14014.
Process 14014 waits for ShareLock on transaction 42981; blocked by process 14021.
Process 14021 waits for ShareLock on transaction 42980; blocked by process 13993.
Process 13993: INSERT INTO mur_global_raw(lat,lon,time,analysed_sst) (SELECT lat,lon,time,analysed_sst FROM mur_global_raw_tmp_75410038 as tmp_vals ORDER BY lat,lon,time,analysed_sst) ON CONFLICT DO NOTHING;
Process 14014: INSERT INTO mur_global_raw(lat,lon,time,analysed_sst) (SELECT lat,lon,time,analysed_sst FROM mur_global_raw_tmp_41473761 as tmp_vals ORDER BY lat,lon,time,analysed_sst) ON CONFLICT DO NOTHING;
Process 14021: INSERT INTO mur_global_raw(lat,lon,time,analysed_sst) (SELECT lat,lon,time,analysed_sst FROM mur_global_raw_tmp_28913605 as tmp_vals ORDER BY lat,lon,time,analysed_sst) ON CONFLICT DO NOTHING;
2020-06-22 23:03:43 UTC:xxxxx(38090):postgres@public_data_raw:[13993]:HINT: See server log for query details.
2020-06-22 23:03:43 UTC:xxxxx(38090):postgres@public_data_raw:[13993]:CONTEXT: while inserting index tuple (1969403,34) in relation "mur_global_raw"
2020-06-22 23:03:43 UTC:xxxxx(38090):postgres@public_data_raw:[13993]:STATEMENT: INSERT INTO mur_global_raw(lat,lon,time,analysed_sst) (SELECT lat,lon,time,analysed_sst FROM mur_global_raw_tmp_75410038 as tmp_vals ORDER BY lat,lon,time,analysed_sst) ON CONFLICT DO NOTHING;

这些死锁持续且定期发生，因此希望我可以解决设计中的某个组件来避免它们。我对正在发生的锁的理解显然不足以解决现阶段的问题。

如果有人能帮助我理解导致这种三向死锁的锁和事务，我将不胜感激。当然，如果您有如何避免它的想法，我也欢迎。

衷心感谢 SO 社区。​​p>

Answer 1

我最好的解决方法是在启动 upsert 之前添加一个独占锁，如下所示：

LOCK TABLE mur_global_raw IN EXCLUSIVE MODE;

欢迎任何 cmets。

Answer 2

如果您找不到更好的方法，请捕获死锁错误并重复事务。如果死锁经常发生，那会很烦人并且会损害性能，但它比表锁要好，因为它不会阻止 autovacuum 完成它的重要工作。

也许您可以减少批处理的大小或持续时间以减少死锁的可能性。