在 csv 导入期间在 Postgres 中处理最大 1600 列

Question

我正在尝试将大量 csv 文件（近 10000 列）导入 postgres 数据库。通常要导入文件，我会将其复制到临时表中，然后使用该临时表将所有内容放在正确的位置。但是 postgres 的最大列数为 1600，因此不会导入 csv。我无法控制 csv 的大小或它的格式，所以我需要使用它。

有没有办法增加临时表的这个值，或者使用复制命令解析成多个临时表？如果您有任何建议，我也可以使用其他方式导入文件。

有什么建议吗？我目前正在使用 Copy 命令导入 csv：

COPY INTAKE
FROM 'file location'
CSV HEADER;

感谢您的帮助！

Answer 1

将其全部导入为单个列，如下所述：

https://stackoverflow.com/a/60120879/16361

一旦有了这些，您就可以组合一个查询，该查询将生成 SQL 以按照您想要的方式提取您想要的字段。一个有 5 列且最大表大小为 2 列的小示例（IRL，您希望在此代码块之后的代码块中将 max_cols_per_table 设置为较大的数字，例如 1kish，因此您要创建 10 个表而不是 5000 个）。

$ cat /tmp/fivewide.csv
c1,c2,c3,c4,c5
r1c1,r1c2,r1c3,r1c4,r1c5
r2c1,r2c2,r2c3,r2c4,r2c5
$ psql -X testdb
psql (12.3 (Debian 12.3-1), server 10.5 (Debian 10.5-1+build4))
Type "help" for help.

testdb=# create table my_import_table(data text);
CREATE TABLE
testdb=# \copy my_import_table from /tmp/fivewide.csv csv delimiter e'\x01' quote e'\x02'
COPY 3
testdb=# select * from my_import_table;
           data           
--------------------------
 c1,c2,c3,c4,c5
 r1c1,r1c2,r1c3,r1c4,r1c5
 r2c1,r2c2,r2c3,r2c4,r2c5
(3 rows)

生成 CREATE TABLE 语句，以便将其拆分为实际列（如果值中有逗号，这将中断；我不会在 SQL 中实现完整的 CSV 解析器 :)

testdb=# with
settings as (select 2 as max_cols_per_table, 'my_import_table' as import_table, 'data' as column_name),
computed_settings1 as (select array_length(string_to_array(d, ','), 1) as num_cols from t limit 1),
computed_settings2 as (select (ceil((select num_cols::float from computed_settings1) / (select max_cols_per_table from settings)))::integer as num_tables),
columns_exprs as (select i, '(string_to_array('||(select column_name from settings)||$$, ','))[$$||i||'] AS col'||i as cexpr from generate_series(1, (select num_cols from computed_settings1)) as i),
column_exprs_by_table as (select t, cexpr from generate_series(1, (select num_tables from computed_settings2)) as t join columns_exprs ce
on i>((t-1)*(select max_cols_per_table from settings))
AND i<=(t*(select max_cols_per_table from settings))
),
create_table_stmts as (select 'create table t_'||t||' AS SELECT '||string_agg(cexpr, ', ')||' FROM '||(select import_table from settings)||';' from column_exprs_by_table group by t)
select * from create_table_stmts;
                                                             ?column?                                                              
-----------------------------------------------------------------------------------------------------------------------------------
 create table t_3 AS SELECT (string_to_array(data, ','))[5] AS col5 FROM my_import_table;
 create table t_2 AS SELECT (string_to_array(data, ','))[3] AS col3, (string_to_array(data, ','))[4] AS col4 FROM my_import_table;
 create table t_1 AS SELECT (string_to_array(data, ','))[1] AS col1, (string_to_array(data, ','))[2] AS col2 FROM my_import_table;
(3 rows)

使用DO 执行此 DDL 并检查结果：

testdb=# DO $outer$
DECLARE
stmt text;
BEGIN
FOR stmt IN
with
settings as (select 2 as max_cols_per_table, 'my_import_table' as import_table, 'data' as column_name),
computed_settings1 as (select array_length(string_to_array(d, ','), 1) as num_cols from t limit 1),
computed_settings2 as (select (ceil((select num_cols::float from computed_settings1) / (select max_cols_per_table from settings)))::integer as num_tables),
columns_exprs as (select i, '(string_to_array('||(select column_name from settings)||$$, ','))[$$||i||'] AS col'||i as cexpr from generate_series(1, (select num_cols from computed_settings1)) as i),
column_exprs_by_table as (select t, cexpr from generate_series(1, (select num_tables from computed_settings2)) as t join columns_exprs ce
on i>((t-1)*(select max_cols_per_table from settings))
AND i<=(t*(select max_cols_per_table from settings))
),
create_table_stmts as (select 'create table t_'||t||' AS SELECT '||string_agg(cexpr, ', ')||' FROM '||(select import_table from settings)||';' from column_exprs_by_table group by t)
select * from create_table_stmts
LOOP
EXECUTE stmt;
END LOOP;
END;
$outer$;
DO
testdb=# select * from t_1;
 col1 | col2 
------+------
 c1   | c2
 r1c1 | r1c2
 r2c1 | r2c2
(3 rows)

testdb=# select * from t_2;
 col3 | col4 
------+------
 c3   | c4
 r1c3 | r1c4
 r2c3 | r2c4
(3 rows)

testdb=# select * from t_3;
 col5 
------
 c5
 r1c5
 r2c5
(3 rows)

进一步的改进，也许是聪明地使用第 1 行作为列名，或者总是包含一些对连接表有用的列，留给读者作为练习。