hadoop2.3 eclipse提交任务到集群上 找不到input

battier 发布于 2016/01/10 13:50
阅读 305
收藏 0

问题描述:想在window环境下通过eclipse将统计字数job提交给虚拟机上的集群,但是异常,请各位指点迷津

问题截图:集群是正常的,如图:


(这是在管理节点上的进程,datanode在另外两台机器上,也是正常的)

hdfs input文件夹也是正常的,如图:


在eclipse上执行job 报错信息:

2016-01-10 13:36:15,643 WARN  [main] security.UserGroupInformation (UserGroupInformation.java:doAs(1551)) - PriviledgedActionException as:hadoop (auth:SIMPLE) cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://192.168.88.128:9000/hadoop/input
Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://192.168.88.128:9000/hadoop/input
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:285)
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:340)
	at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493)
	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313)
	at com.battier.hadoop.test.WordCount.main(WordCount.java:145)

HDFS也是可以正常访问的:



执行的字数统计代码:

/**
	         * 连接hadoop集群配置
	         */
			conf.set("dfs.permissions","false");
	        //conf.set("fs.default.name", "hdfs://192.168.88.128:9000");
			conf.set("fs.defaultFS", "hdfs://hadoop01:9000");
	        conf.set("hadoop.job.user", "hadoop");
	        conf.set("mapreduce.framework.name", "yarn");
	        conf.set("mapreduce.jobtracker.address", "192.168.88.128:9001");
	        conf.set("yarn.resourcemanager.hostname", "192.168.88.128");
	        conf.set("yarn.resourcemanager.admin.address", "192.168.88.128:8033");
	        conf.set("yarn.resourcemanager.address", "192.168.88.128:8032");
	        conf.set("yarn.resourcemanager.resource-tracker.address", "192.168.88.128:8031");
	        conf.set("yarn.resourcemanager.scheduler.address", "192.168.88.128:8030");
	 
	        String[] otherArgs = new String[2];
	        otherArgs[0] = "hdfs://hadoop01:9000/hadoop/input";//计算原文件目录,需提前在里面存入文件
	        String time = new SimpleDateFormat("yyyyMMddHHmmss").format(new Date());
	        otherArgs[1] = "hdfs://hadoop01:9000/hadoop/output/" + time;//计算后的计算结果存储目录,每次程序执行的结果目录不能相同,所以添加时间标签



请指教,感谢!


加载中
0
飞弹蛋
飞弹蛋

帮顶,是不是

Input path does not exist: hdfs://192.168.88.128:9000/hadoop/input

battier
battier
感谢!hadoop01 就是 192.168.88.128
0
Ryan-瑞恩
Ryan-瑞恩

执行 一下 : hadoop fs -ls /hadoop/input   先验证一下你的路径对不对。

路径不存在,,说明,1.你在 hadoop 没有这个路径。2.没有那个文件;

Ryan-瑞恩
Ryan-瑞恩
回复 @battier : 不需要,,,在你的classpath 下面 记得放那三个文件就可以。
battier
battier
感谢指导!hadoop2.x 的输入输出路径要加安装hadoop的用户名作为目录根目录吗?
返回顶部
顶部