mapreduce框架下进行文件存储,reduce任务出错。不知道为什么?

CheckMate 发布于 2013/04/23 21:23
阅读 753
收藏 0

在ubuntu10.04上, hadoop1.1.1, hbase0.92, jdk1.7

本来想写一个在mapreduce下的分布式文件存储。把文件存至HDFS后再提取文件基本信息存入HBase. 结果reduce任务failed了。以下是log:

13/04/23 21:03:46 INFO input.FileInputFormat: Total input paths to process : 2
13/04/23 21:03:46 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/04/23 21:03:46 WARN snappy.LoadSnappy: Snappy native library not loaded
13/04/23 21:03:46 INFO mapred.JobClient: Running job: job_201304232018_0006
13/04/23 21:03:47 INFO mapred.JobClient:  map 0% reduce 0%
13/04/23 21:03:54 INFO mapred.JobClient:  map 100% reduce 0%
13/04/23 21:04:02 INFO mapred.JobClient:  map 100% reduce 33%
13/04/23 21:04:05 INFO mapred.JobClient:  map 100% reduce 100%
13/04/23 21:04:36 INFO mapred.JobClient: Task Id : attempt_201304232018_0006_r_000000_0, Status : FAILED
java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@a8857e closed
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:794)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)
at testforwrite$writeReducer.reduce(testforwrite.java:101)
at testforwrite$writeReducer.reduce(testforwrite.java:1)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.Child.main(Child.java:249)


attempt_201304232018_0006_r_000000_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
attempt_201304232018_0006_r_000000_2: log4j:WARN Please initialize the log4j system properly.
13/04/23 21:06:04 INFO mapred.JobClient:  map 100% reduce 0%
13/04/23 21:06:12 INFO mapred.JobClient:  map 100% reduce 33%
13/04/23 21:06:15 INFO mapred.JobClient:  map 100% reduce 100%
13/04/23 21:06:47 INFO mapred.JobClient:  map 100% reduce 0%
13/04/23 21:06:48 INFO mapred.JobClient: Job complete: job_201304232018_0006
13/04/23 21:06:48 INFO mapred.JobClient: Counters: 23
13/04/23 21:06:48 INFO mapred.JobClient:   Job Counters 
13/04/23 21:06:48 INFO mapred.JobClient:     Launched reduce tasks=4
13/04/23 21:06:48 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=11636
13/04/23 21:06:48 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/04/23 21:06:48 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/04/23 21:06:48 INFO mapred.JobClient:     Launched map tasks=2
13/04/23 21:06:48 INFO mapred.JobClient:     Data-local map tasks=2
13/04/23 21:06:48 INFO mapred.JobClient:     Failed reduce tasks=1
13/04/23 21:06:48 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=172067
13/04/23 21:06:48 INFO mapred.JobClient:   FileSystemCounters
13/04/23 21:06:48 INFO mapred.JobClient:     HDFS_BYTES_READ=1132
13/04/23 21:06:48 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=48496
13/04/23 21:06:48 INFO mapred.JobClient:   File Input Format Counters 
13/04/23 21:06:48 INFO mapred.JobClient:     Bytes Read=914
13/04/23 21:06:48 INFO mapred.JobClient:   Map-Reduce Framework
13/04/23 21:06:48 INFO mapred.JobClient:     Map output materialized bytes=948
13/04/23 21:06:48 INFO mapred.JobClient:     Combine output records=0
13/04/23 21:06:48 INFO mapred.JobClient:     Map input records=11
13/04/23 21:06:48 INFO mapred.JobClient:     Physical memory (bytes) snapshot=288870400
13/04/23 21:06:48 INFO mapred.JobClient:     Spilled Records=11
13/04/23 21:06:48 INFO mapred.JobClient:     Map output bytes=914
13/04/23 21:06:48 INFO mapred.JobClient:     CPU time spent (ms)=720
13/04/23 21:06:48 INFO mapred.JobClient:     Total committed heap usage (bytes)=231350272
13/04/23 21:06:48 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=695681024
13/04/23 21:06:48 INFO mapred.JobClient:     Combine input records=0
13/04/23 21:06:48 INFO mapred.JobClient:     Map output records=11
13/04/23 21:06:48 INFO mapred.JobClient:     SPLIT_RAW_BYTES=218

出错的那行java代码:

HTable table = new HTable(confForBase, Bytes.toBytes("version4"));


怀疑是无法连接hbase. 可是之前在非mapreduce框架下测试过,并没有出现问题。请问大牛们这可能是哪儿出问题了呢?


加载中
返回顶部
顶部