Spring Boot - H2 DB 导入 4GB data.sql OutOfMemory 错误答案

【问题标题】：Spring Boot - H2 DB import 4GB data.sql OutOfMemory ErrorSpring Boot - H2 DB 导入 4GB data.sql OutOfMemory 错误
【发布时间】：2020-05-15 05:23:05
【问题描述】：

我想将我的数据导入到大约 4GB 的 data.sql 中但是 OutOfMemory - 发生堆空间错误。

我已经在 IntelliJ 中增加了堆空间，但仍然无法正常工作。 (-Xms16g -Xmx16g)

我的错误是：

Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_211]
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) ~[na:1.8.0_211]
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) ~[na:1.8.0_211]
    at java.lang.StringBuilder.append(StringBuilder.java:136) ~[na:1.8.0_211]
    at org.springframework.jdbc.datasource.init.ScriptUtils.readScript(ScriptUtils.java:388) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.jdbc.datasource.init.ScriptUtils.readScript(ScriptUtils.java:330) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.jdbc.datasource.init.ScriptUtils.executeSqlScript(ScriptUtils.java:582) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.jdbc.datasource.init.ResourceDatabasePopulator.populate(ResourceDatabasePopulator.java:254) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.jdbc.datasource.init.DatabasePopulatorUtils.execute(DatabasePopulatorUtils.java:49) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer.runScripts(DataSourceInitializer.java:202) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer.initSchema(DataSourceInitializer.java:119) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker.onApplicationEvent(DataSourceInitializerInvoker.java:91) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker.onApplicationEvent(DataSourceInitializerInvoker.java:38) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:403) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:360) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.boot.autoconfigure.orm.jpa.DataSourceInitializedPublisher.publishEventIfRequired(DataSourceInitializedPublisher.java:99) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.boot.autoconfigure.orm.jpa.DataSourceInitializedPublisher.postProcessAfterInitialization(DataSourceInitializedPublisher.java:90) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:431) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$151/1711185459.getObject(Unknown Source) ~[na:na]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1108) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:868) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]

而我的 applicaiton.yml(h2 config) 是：

spring:
  h2:
    console:
      enabled: true
      path: /h2-console

  jpa:
    database: h2
    showSql: false
    hibernate:
      ddl-auto: create
    properties:
      hibernate.dialect: org.hibernate.dialect.H2Dialect
      hibernate.format_sql: true
      hibernate.use_sql_comments: true

  datasource:
    platform: h2
    url: jdbc:h2:mem:testdb;MODE=MYSQL
    username: sa
    password:
    initialization-mode: embedded

我该如何解决这个错误？

【问题讨论】：

如何开始导入？使用 Intellij 中的 Run 按钮或通过命令行？
您运行的是 32 位还是 64 位 JVM？ stackoverflow.com/questions/3030263/…
@GabLeg 我使用 Intellij 和 64 位 JVM 的 Run 按钮运行它
@hanswsw 你应该很好。我会检查java.util.Arrays.copyOf(Arrays.java:3332)。你可能会得到heap space，即使是 16g，因为副本太多。
您是否尝试在数据源初始化本身上使用 H2 RunScript？正如@GabLeg 指出的那样，即使是 16G 的分配也可能不够。您可能需要配置文件以观察实际需求。

标签： spring spring-boot jvm heap-memory intellij-14

【解决方案1】：

tl;dr：spring 数据库 sql 初始化并不意味着用于大文件

OutOfMemory 来自您对 spring 数据库 sql 初始化的使用

https://docs.spring.io/spring-boot/docs/current/reference/html/howto.html#howto-initialize-a-database-using-spring-jdbc

它不是为了执行这么大的文件。

Spring 会首先将文件加载到内存中在加载过程中，文件被逐行添加到 StringBuilder 中，该 StringBuilder 会不断重新分配，从而导致内存分配出现临时峰值

见https://github.com/spring-projects/spring-framework/blob/0ddc62e87f9e0c9fb664ba085bcdf7ca61e697dd/spring-jdbc/src/main/java/org/springframework/jdbc/datasource/init/ScriptUtils.java#L380-L392

根据您的 jvm 版本，将文件存储在内存中所需的内存可能比原始文件大小大很多（因为它可能以 unicode 存储）

然后将生成的StringBuilder转换为String（临时内存分配）

然后字符串被 sql 语句分割，全部添加到数组列表中（脚本所需的内存加倍）

见https://github.com/spring-projects/spring-framework/blob/0ddc62e87f9e0c9fb664ba085bcdf7ca61e697dd/spring-jdbc/src/main/java/org/springframework/jdbc/datasource/init/ScriptUtils.java#L596

【讨论】：

【解决方案2】：

转到终端并使用此命令启动您的应用程序

java -jar your_app_name.jar -Xms16g -Xmx16g

【讨论】：

我想从 intellij 运行它，我还给了 -Xms16g -Xmx16g VM 参数，但没有成功。