Spark任务提交错误

rilweic 发布于 2016/05/04 10:19
阅读 1K+
收藏 0
采用cloudera manager 5.4.5安装完spark后提交PI作业报错

提交命令

spark-submit --class org.apache.spark.examples.SparkPi  --master yarn-cluster   /opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/spark/lib/spark-examples-1.3.0-cdh5.4.5-hadoop2.6.0-cdh5.4.5.jar 10


java.io.FileNotFoundException: /user/spark/applicationHistory/application_1462236388531_0063.inprogress (没有那个文件或目录)
	at java.io.FileOutputStream.open(Native Method)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:110)
	at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:117)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:399)
	at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28)
	at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)




                
加载中
0
a
akzhao
报错的第一句不就已经告诉你原因了吗
rilweic
rilweic
不是,默认是在hdfs上的路径,但是CDH找本地去了。后来查配置发现只有namenode上有hdfs的标识,其他的都没有。手动修改以后成功运行
返回顶部
顶部