spark-submit yarn-client 运行失败

starmichael 发布于 2015/08/26 17:01
阅读 4K+
收藏 0
2015-08-26 16:48:55,941 INFO  [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing view acls to: root
2015-08-26 16:48:55,946 INFO  [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing modify acls to: root
2015-08-26 16:48:55,947 INFO  [main] spark.SecurityManager (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
2015-08-26 16:48:56,403 INFO  [sparkDriver-akka.actor.default-dispatcher-4] slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
2015-08-26 16:48:56,456 INFO  [sparkDriver-akka.actor.default-dispatcher-4] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
2015-08-26 16:48:56,816 INFO  [sparkDriver-akka.actor.default-dispatcher-5] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening on addresses :[akka.tcp://sparkDriver@hadoop05:54033]
2015-08-26 16:48:56,820 INFO  [sparkDriver-akka.actor.default-dispatcher-5] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting now listens on addresses: [akka.tcp://sparkDriver@hadoop05:54033]
2015-08-26 16:48:56,832 INFO  [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'sparkDriver' on port 54033.
2015-08-26 16:48:56,857 INFO  [main] spark.SparkEnv (Logging.scala:logInfo(59)) - Registering MapOutputTracker
2015-08-26 16:48:56,870 INFO  [main] spark.SparkEnv (Logging.scala:logInfo(59)) - Registering BlockManagerMaster
2015-08-26 16:48:56,889 INFO  [main] storage.DiskBlockManager (Logging.scala:logInfo(59)) - Created local directory at /tmp/spark-local-20150826164856-d539
2015-08-26 16:48:56,893 INFO  [main] storage.MemoryStore (Logging.scala:logInfo(59)) - MemoryStore started with capacity 530.3 MB
2015-08-26 16:48:57,347 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-08-26 16:48:57,538 INFO  [main] spark.HttpFileServer (Logging.scala:logInfo(59)) - HTTP File server directory is /tmp/spark-8ea80eb5-5ae9-4f11-8c5f-4616d14fc3d3
2015-08-26 16:48:57,543 INFO  [main] spark.HttpServer (Logging.scala:logInfo(59)) - Starting HTTP Server
2015-08-26 16:48:57,603 INFO  [main] server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
2015-08-26 16:48:57,624 INFO  [main] server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started SocketConnector@0.0.0.0:50271
2015-08-26 16:48:57,624 INFO  [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'HTTP file server' on port 50271.
2015-08-26 16:48:57,739 INFO  [main] server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
2015-08-26 16:48:57,752 INFO  [main] server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started SelectChannelConnector@0.0.0.0:4040
2015-08-26 16:48:57,753 INFO  [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'SparkUI' on port 4040.
2015-08-26 16:48:57,756 INFO  [main] ui.SparkUI (Logging.scala:logInfo(59)) - Started SparkUI at http://hadoop05:4040
2015-08-26 16:48:57,885 INFO  [main] spark.SparkContext (Logging.scala:logInfo(59)) - Added JAR /home/hbase/cdrp-0.0.2-SNAPSHOT.jar at http://10.2.98.144:50271/jars/cdrp-0.0.2-SNAPSHOT.jar with timestamp 1440578937884
2015-08-26 16:48:57,886 INFO  [main] spark.SparkContext (Logging.scala:logInfo(59)) - Added JAR /home/hbase/oozie-client-4.0.0-cdh5.2.0.jar at http://10.2.98.144:50271/jars/oozie-client-4.0.0-cdh5.2.0.jar with timestamp 1440578937886
2015-08-26 16:48:58,052 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at /0.0.0.0:8032
2015-08-26 16:48:58,283 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Requesting a new application from cluster with 7 NodeManagers
2015-08-26 16:48:58,298 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Verifying our application has not requested more than the maximum memory capability of the cluster (9990 MB per container)
2015-08-26 16:48:58,299 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Will allocate AM container, with 1408 MB memory including 384 MB overhead
2015-08-26 16:48:58,300 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Setting up container launch context for our AM
2015-08-26 16:48:58,305 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Preparing resources for our AM container
2015-08-26 16:48:58,442 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Source and destination file systems are the same. Not copying file:/opt/cloudera/parcels/CDH-5.3.5-1.cdh5.3.5.p0.4/jars/spark-assembly-1.2.0-cdh5.3.5-hadoop2.5.0-cdh5.3.5.jar
2015-08-26 16:48:58,526 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Setting up the launch environment for our AM container
2015-08-26 16:48:58,731 INFO  [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing view acls to: root
2015-08-26 16:48:58,732 INFO  [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing modify acls to: root
2015-08-26 16:48:58,732 INFO  [main] spark.SecurityManager (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
2015-08-26 16:48:58,735 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Submitting application 42 to ResourceManager
2015-08-26 16:48:58,776 INFO  [main] impl.YarnClientImpl (YarnClientImpl.java:submitApplication(236)) - Submitted application application_1440484199657_0042
2015-08-26 16:48:59,782 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED)
2015-08-26 16:48:59,790 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: root.root
         start time: 1440578938748
         final status: UNDEFINED
         tracking URL: http://hadoop05:8088/proxy/application_1440484199657_0042/
         user: root
2015-08-26 16:49:00,794 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED)
2015-08-26 16:49:01,798 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED)
2015-08-26 16:49:02,803 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED)
2015-08-26 16:49:03,807 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED)
2015-08-26 16:49:04,812 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED)
2015-08-26 16:49:05,818 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED)
2015-08-26 16:49:06,823 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED)
2015-08-26 16:49:07,828 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED)
2015-08-26 16:49:08,832 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED)
2015-08-26 16:49:09,837 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED)
2015-08-26 16:49:10,844 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED)
2015-08-26 16:49:11,849 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED)
2015-08-26 16:49:12,853 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: FAILED)
2015-08-26 16:49:12,855 INFO  [main] yarn.Client (Logging.scala:logInfo(59)) - 
         client token: N/A
         diagnostics: Application application_1440484199657_0042 failed 2 times due to AM Container for appattempt_1440484199657_0042_000002 exited with  exitCode: 1 due to: Exception from container-launch.
Container id: container_1440484199657_0042_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
        at org.apache.hadoop.util.Shell.run(Shell.java:455)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)




Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: root.root
         start time: 1440578938748
         final status: FAILED
         tracking URL: http://hadoop05:8088/cluster/app/application_1440484199657_0042
         user: root
Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:102)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:140)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:335)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
        at com.goodwill.cdr.hadoop.spark.PatientCountJHMK.execute(PatientCountJHMK.java:56)
        at com.goodwill.cdr.hadoop.spark.PatientCountJHMK.main(PatientCountJHMK.java:40)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

以下是问题补充:

@starmichael:spark-submit --class com.cdr.hadoop.spark.PatientCountJHMK \ --master yarn \ --deploy-mode client \ --executor-memory 512m\ --driver-memory 1g \ --num-executors 18 \ cdrp-0.0.2-SNAPSHOT.jar (2015/08/26 17:03)
@starmichael:CDH集群5.2.0升级到5.3.5,我把相关jar包升级后,尝试跑原来的任务,报错 (2015/08/26 17:06)
加载中
返回顶部
顶部