cloudera Oozie sqoop2 作业永远挂起心跳心跳答案

【问题标题】：cloudera Oozie sqoop2 job hangs running forever Heart beat Heart beatcloudera Oozie sqoop2 作业永远挂起心跳心跳
【发布时间】：2017-02-26 21:05:42
【问题描述】：

我正在尝试使用 oozie 并行运行两个 sqoop 作业。但是有两个工作在 95% 之后就卡住了，另外两个处于接受状态。我还增加了纱线资源最大内存。还加了

<name>mapred.tasktracker.map.tasks.maximum</name>
<value>50 </value> 
 in mapred-site.xml , but nothing helped. please help.

Yarn Cluster Metrix:
Apps Submitted          4
Apps Pending            2
Apps Running            2
Apps Completed          0
Containers Running      4
Memory Used             10GB
Memory Total            32GB
Memory Reserved         0B
VCores Used             4
VCores Total            24
VCores Reserved         0
Active Nodes            4
Decommissioned Nodes    0
Lost Nodes              0
Unhealthy Nodes         0
Rebooted Nodes          0
----------
Sysout Log 
========================================================================
3175 [main] WARN  org.apache.sqoop.tool.SqoopTool  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
  3198 [main] INFO  org.apache.sqoop.Sqoop  - Running Sqoop version: 1.4.5-cdh5.2.0
  3212 [main] WARN  org.apache.sqoop.tool.BaseSqoopTool  - Setting your password on the command-line is insecure. Consider using -P instead.
  3213 [main] INFO  org.apache.sqoop.tool.BaseSqoopTool  - Using Hive-specific delimiters for output. You can override
  3213 [main] INFO  org.apache.sqoop.tool.BaseSqoopTool  - delimiters with --fields-terminated-by, etc.
  3224 [main] WARN  org.apache.sqoop.ConnFactory  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
  3280 [main] INFO  org.apache.sqoop.manager.oracle.OraOopManagerFactory  - Data Connector for Oracle and Hadoop is disabled.
  3293 [main] INFO  org.apache.sqoop.manager.SqlManager  - Using default fetchSize of 1000
  3297 [main] INFO  org.apache.sqoop.tool.CodeGenTool  - Beginning code generation
  3951 [main] INFO  org.apache.sqoop.manager.OracleManager  - Time zone has been set to GMT
  4023 [main] INFO  org.apache.sqoop.manager.SqlManager  - Executing SQL statement: SELECT t.* FROM PT_PRELIM_FINDING_V t WHERE 1=0
  4068 [main] INFO  org.apache.sqoop.orm.CompilationManager  - HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hadoop-mapreduce
  5925 [main] INFO  org.apache.sqoop.orm.CompilationManager  - Writing jar file: /tmp/sqoop-nobody/compile/0dab11f6545d8ef69d6dd0f6b9041a50/PT_PRELIM_FINDING_CYTOGEN_V.jar
  5937 [main] INFO  org.apache.sqoop.mapreduce.ImportJobBase  - Beginning import of PT_PRELIM_FINDING_V
  5962 [main] INFO  org.apache.sqoop.manager.OracleManager  - Time zone has been set to GMT
  5981 [main] WARN  org.apache.sqoop.mapreduce.JobBase  - SQOOP_HOME is unset. May not be able to find all job dependencies.
  6769 [main] INFO  org.apache.sqoop.mapreduce.db.DBInputFormat  - Using read commited transaction isolation
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat 
  Heart beat

【问题讨论】：

您的数据库似乎没有响应。也许你需要在那里增加并行化。如果您联系 user@sqoop.apache.org 邮件列表，社区可能会为您提供帮助。

标签： sqoop hadoop-yarn oozie hadoop2 cloudera-cdh

【解决方案1】：

感谢@abeaamase。

我要求我们的 DBA 将 oracle 数据库最大进程增加到 750，将最大会话池增加到大约 1.5 倍进程大小，即 1125。这已经解决了这个问题。这与纱线内存无关。不幸的是在sqoop2中没有处理这个异常。如果您觉得此解释不恰当，请随时添加更多答案。

【讨论】：