从eclipse提交到spark遇到问题

xiaojiongen123 发布于 2015/10/11 23:00
阅读 1K+
收藏 0

我从eclipse 直接提交job到spark,一直显示如下的错误:15/10/11 14:47:51 INFO DAGScheduler: Registering RDD 3 (mapToPair at JavaWordCount.java:55)

15/10/11 14:47:51 INFO DAGScheduler: Got job 0 (collect at JavaWordCount.java:69) with 1 output partitions (allowLocal=false)
15/10/11 14:47:51 INFO DAGScheduler: Final stage: Stage 1(collect at JavaWordCount.java:69)
15/10/11 14:47:51 INFO DAGScheduler: Parents of final stage: List(Stage 0)
15/10/11 14:47:51 INFO DAGScheduler: Missing parents: List(Stage 0)
15/10/11 14:47:51 INFO DAGScheduler: Submitting Stage 0 (MapPartitionsRDD[3] at mapToPair at JavaWordCount.java:55), which has no missing parents
15/10/11 14:47:51 INFO MemoryStore: ensureFreeSpace(4344) called with curMem=153973, maxMem=991470551
15/10/11 14:47:51 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.2 KB, free 945.4 MB)
15/10/11 14:47:51 INFO MemoryStore: ensureFreeSpace(3118) called with curMem=158317, maxMem=991470551
15/10/11 14:47:51 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.0 KB, free 945.4 MB)
15/10/11 14:47:51 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on sandbox.hortonworks.com:53261 (size: 3.0 KB, free: 945.5 MB)
15/10/11 14:47:51 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
15/10/11 14:47:51 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:839
15/10/11 14:47:51 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (MapPartitionsRDD[3] at mapToPair at JavaWordCount.java:55)
15/10/11 14:47:51 INFO YarnScheduler: Adding task set 0.0 with 1 tasks
15/10/11 14:48:07 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/11 14:48:22 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/11 14:48:37 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/11 14:48:52 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/11 14:49:07 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/11 14:49:22 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/11 14:49:37 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources


一直在这个状态下不动了,求高手解救?


加载中
0
李云聪
李云聪

我有两个办法:

一、使你的eclipse中引用的jar版本和集群版本不一致(重点看看你的scala版本,官方发布的spark用的是2.10编译的)

二.在集群上配置你的机子的ip和主机名

返回顶部
顶部