Hive - 内存不足异常

Question

这导致了 MR 工作。 MR 作业成功运行，但是当蜂蜡尝试渲染结果时，我得到了 OOM 异常。

我想知道是否有配置设置可以帮助我解决这个问题。

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Arrays.java:2694)
    at java.lang.String.<init>(String.java:203)
    at java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:561)
    at java.nio.CharBuffer.toString(CharBuffer.java:1201)
    at org.apache.hadoop.io.Text.decode(Text.java:394)
    at org.apache.hadoop.io.Text.decode(Text.java:371)
    at org.apache.hadoop.io.Text.toString(Text.java:273)
    at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:280)
    at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:220)
    at org.apache.hadoop.hive.serde2.DelimitedJSONSerDe.serializeField(DelimitedJSONSerDe.java:59)
    at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:427)
    at org.apache.hadoop.hive.ql.exec.ListSinkOperator.processOp(ListSinkOperator.java:91)
    at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
    at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:498)
    at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:137)
    at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1474)
    at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.materializeResults(BeeswaxServiceImpl.java:434)
    at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.fetch(BeeswaxServiceImpl.java:543)
    at com.cloudera.beeswax.BeeswaxServiceImpl$5.run(BeeswaxServiceImpl.java:986)
    at com.cloudera.beeswax.BeeswaxServiceImpl$5.run(BeeswaxServiceImpl.java:981)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
    at com.cloudera.beeswax.BeeswaxServiceImpl.doWithState(BeeswaxServiceImpl.java:772)
    at com.cloudera.beeswax.BeeswaxServiceImpl.fetch(BeeswaxServiceImpl.java:980)
    at com.cloudera.beeswax.api.BeeswaxService$Processor$fetch.getResult(BeeswaxService.java:987)
    at com.cloudera.beeswax.api.BeeswaxService$Processor$fetch.getResult(BeeswaxService.java:971)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

更新 我增加了 cloudera manager 中的内存设置，但没有雪茄。在我第一次运行查询时重新启动服务后，它可以工作。第二次运行失败：

• Hue - 蜂蜡服务器（默认）/资源管理 - Java 堆大小 Beeswax 服务器的字节数 [1 Gib]
• Hive - 网关（默认）/资源管理 - 客户端 Java 堆大小（以字节为单位）[1 Gib]
• Hive - HiveServer2 （默认）/Resource Management - Java Heap Size of HiveServer2 in 字节 [1 Gib]

Answer 1

您可以使用（增加）三个 -Xmx - 客户端 java、Hive Server 2 和 Hive Meta Store Server。我猜你正在达到这些限制之一。