更快地批量加载嵌套数据

Question

我正在填充两个表，它们具有一对多的关系。

所以我在outer 中插入一行，获取该行的（自动增量主键）id，然后在inner 中插入 100 行（全部带有指向outer.id 的外键）。

然后我重复 50 次。对于outer 中的每个条目，我必须插入、读取id，然后插入到内部。

这很慢。大部分时间都花在将 100 行加载到inner 中。我怀疑如果我可以在一个批处理操作中将所有 50*100 行插入inner，它会快得多。但我不知道该怎么做 - 我怎样才能让外键工作？

其他人如何提高效率？

我正在使用 Java / Spring。这 100 行插入了 JdbcTemplate.batchUpdate()。

public final void insert(final JdbcTemplate db,
                         final Iterable<DataBlock> data) {
    String insertSql = getInsertSql();
    String idQuery = getIdQuery();
    ItemRowMapper.IdRowMapper mapper = new ItemRowMapper.IdRowMapper();
    for (DataBlock block: data) {
        Object[] outer = block.getOuter();
        LOG.trace("Loading outer");
        db.update(insertSql, outer);
        LOG.trace("Getting index");
        // currently retrieve index based on natural key, but could use last index
        int id = db.query(idQuery, mapper, uniqueData(outer)).get(0);
        LOG.trace("Getting inner");
        List<Object[]> inner = block.getInner(id);
        // most time spent here
        LOG.trace(format("Loading inner (%d)", inner.size()));
        innerTable.insert(db, inner);
    }
}

和伪SQL：

create table outer (
    integer id primary key autoincrement,
    ...
);

create table inner (
    integer outer references outer(id),
    ...
);

更新 - 以下内容似乎适用于 Spring 3.1.1 和 Postgres 9.2-1003.jdbc4。

/**
 * An alternative implementation that should be faster, since it inserts
 * in just two batches (one for inner and one fo router).
 *
 * @param db A connection to the database.
 * @param data The data to insert.
 */
public final void insertBatchier(final JdbcTemplate db,
                                 final AllDataBlocks data) {
    final List<Object[]> outers = data.getOuter();
    List<Integer> ids = db.execute(
            new PreparedStatementCreator() {
                @Override
                public PreparedStatement createPreparedStatement(
                        final Connection con) throws SQLException {
                    return con.prepareStatement(getInsertSql(),
                            Statement.RETURN_GENERATED_KEYS);
                }
            },
            new PreparedStatementCallback<List<Integer>>() {
        @Override
        public List<Integer> doInPreparedStatement(final PreparedStatement ps)
                throws SQLException {
            for (Object[] outer: outers) {
                for (int i = 0; i < outer.length; ++i) {
                    setParameterValue(ps, i + 1,
                            SqlTypeValue.TYPE_UNKNOWN, outer[i]);
                }
                ps.addBatch();
            }
            ps.executeBatch();
            RowMapperResultSetExtractor<Integer> ids =
                    new RowMapperResultSetExtractor<Integer>(
                            new ItemRowMapper.IdRowMapper());
            try (ResultSet keys = ps.getGeneratedKeys()) {
                return ids.extractData(keys);
            }
        }
    });
    innerTable.insert(db, data.getInner(ids));
}

Answer 1

我对@9​​87654322@ 不太熟悉，但假设它类似于 JDBC，我会使用与以下代码类似的东西（我可能会将其分解为多种方法）：

private static final int BATCH_SIZE = 50;

public void addBatch(Connection connection, List<Outer> outers) {

  PreparedStatement outerInsertStatement = connection.prepareStatement("...", Statement.RETURN_GENERATED_KEYS);
  PreparedStatement innerInsertStatement = connection.prepareStatement("...", Statement.RETURN_GENERATED_KEYS);

  List<Integer> outerIds = new ArrayList<Integer>();

  for(Outer outer : outers) {
    outerInsertStatement.setParameter(...);
    ...
    outerInsertStatement.setParameter(...);

    outerInsertStatement.addBatch();
  }

  outerInsertStatement.executeBatch();
  //Note, this line requires JDBC3
  ResultSet primaryKeys = outerInsertStatement.getGeneratedKeys();
  while(!primaryKeys.isAfterLast()) {
    outerIds.add(primaryKeys.getInt(0));
  }

  for(int i = 0; i < outers.size(); i++) {
    Outer outer = outers.get(i);
    Integer outerId = outerIds.get(i);
    for(Inner inner : outer.getInners()) {
      //One of these setParameter calls would use outerId
      innerInsertStatement.setParameter(...);
      ...
      innerInsertStatement.setParameter(...);
      innerInsertStatement.addBatch();

      if( (i+1) % BATCH_SIZE == 0) {
        innerInsertStatement.executeBatch();
      }
    }
    innerInsertStatement.executeBatch();
  }
}