在spark中遇到org.apache.spark.SparkException:该如何解决问题

AuroraTan 发布于 2017/09/21 21:19
阅读 633
收藏 0

运行大量数据集就会遇到问题,怎么办???

 

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 33322.0 failed 1 times, most recent failure: Lost task 0.0 in stage 33322.0 (TID 925, localhost): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 159469 ms

Driver stacktrace:

    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)

    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)

    at scala.Option.foreach(Option.scala:236)

    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)

    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1952)

    at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:1025)

    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)

    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)

    at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)

    at org.apache.spark.rdd.RDD.reduce(RDD.scala:1007)

    at org.apache.spark.graphx.impl.VertexRDDImpl.count(VertexRDDImpl.scala:90)

    at com.dbcs.CommunityUtil.communityInfluence(CommunityUtil.scala:427)

    at com.dbcs.CommunityUtil$$anonfun$execute$1.apply(CommunityUtil.scala:542)

    at com.dbcs.CommunityUtil$$anonfun$execute$1.apply(CommunityUtil.scala:530)

    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

    at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:168)

    at com.dbcs.CommunityUtil.execute(CommunityUtil.scala:530)

    at com.dbcs.LouvainHarness.run(LouvainHarness.scala:82)

    at com.dbcs.Main$.main(Main.scala:137)

    at com.dbcs.Main.main(Main.scala)

17/09/21 11:32:28 WARN QueuedThreadPool: 5 threads could not be stopped

加载中
返回顶部
顶部