hive任务转为mapreduce任务后,运行失败了,mr自动kill掉了

余子Gam 发布于 2018/09/03 16:48
阅读 2K+
收藏 0

OSC年度开源问卷新鲜出炉,你的回答对我们很重要,参与开源可以从这份问卷开始>>>

 

配置:centos7.3 ,1核8G内存,hadoop版本2.7.3,hive版本0.12

-----------------------------分割线---------------

具体代码:

        sql = "select count(1) from " + tableName;
		System.out.println("Running:" + sql);
		res = stmt.executeQuery(sql);
		System.out.println("执行“regular hive query”运行结果:");
		while (res.next()) {
			System.out.println("count ------>" + res.getString(1));
		}

 

反馈为:

java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
        at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:165)
        at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:153)
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:198)
        at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:300)
        at com.jhao.hive.HiveJdbcCli.countData(HiveJdbcCli.java:80)
        at com.jhao.hive.HiveJdbcCli.main(HiveJdbcCli.java:49)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)

就是mapreduce任务出问题了。

在运行该程序的同时,另一个窗口打印了部分运行日志:

Table default.person stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 151, raw_data_size: 0]
OK
OK
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_1535958380729_0001, Tracking URL = http://VM_0_13_centos:8088/proxy/application_1535958380729_0001/
Kill Command = /opt/hadoop/bin/hadoop job  -kill job_1535958380729_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2018-09-03 16:04:25,502 Stage-1 map = 0%,  reduce = 0%
Ended Job = job_1535958380729_0001 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Job 0: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

 

嗯,就是mapreduce任务出问题之后被kill掉咯。(根据这个提示应该是这样的)

于是我就去yarn的web页面找找这个任务执行的一些具体细节,如下图。

 

点进去再看看: 是这样的↓↓↓↓↓

 

 

我看了一下那个diagnostics. 这里说了一些问题,(但我没理解啊。。“运行了两次失败了。。blabla") 这是什么原原因??

说明:hadoop是单机式的。

补充一下hadoop我的四个配置文件(hdfs-site.xml,core-site.xml,mapred-site.xml,yarn-site.xml)如下:

core-site.xml的conf标签下内容为:

<configuration>
<property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/hadoop/tmp</value>
    </property>

</configuration>

yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
<property> 
      <name>yarn.nodemanager.aux-services</name> 
      <value>mapreduce_shuffle</value> 
   </property>

</configuration>

mapred-site.xml

<configuration>
<property> 
      <name>mapreduce.framework.name</name> 
      <value>yarn</value> 
   </property>

</configuration>

hdfs-site.xml:

<configuration>
	
	    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.name.dir</name>
        <value>/usr/local/hadoop/hdfs/name</value>
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/usr/local/hadoop/hdfs/data</value>
    </property>
	
<property>
<name>dfs.support.append</name>
<value>true</value>
</property>

<property>
<name>dfs.namenode.rpc-address.ns1.nn1</name>
<value>服务器IP:9000</value>
</property>

</configuration>

 

 

虽然我知道可以通过换sql引擎tez来做同样的事情,但是我想知道现在用mr这个该如何解决?现在这个错误是发生在哪?感谢帮助。

加载中
0
h
hhh111aa
该评论暂时无法显示,详情咨询 QQ 群:912889742
0
z
zb38720246

我的也是  无报错  就是执行mr不可以 本地模式就可以

OSCHINA
登录后可查看更多优质内容
返回顶部
顶部