通过spark读取hbase数据线程池拒绝连接

shining027 发布于 2017/06/16 09:40
阅读 86
收藏 0

借助Hive的HBaseTableInputFormat建表然后在sparksql中分析hbase表数据,10000条数据在虚拟机上可以正常运行,在物理机上10w条数据却报异常,错误信息如下:

 org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 1.0 failed 1 times, most recent failure: Lost task 2.0 in stage 1.0 (TID 3, localhost): java.lang.RuntimeException: java.util.concurrent.                                                  : Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFutureQueueingFuture@37b2d978 rejected from java.util.concurrent.ThreadPoolExecutor@46477dd0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 14109]

    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)

    at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)

    at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:403)

    at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)

    at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:205)

    at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:147)

    at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:216)

    at 

加载中
返回顶部
顶部