关于hadoop hdfs文件丢失问题?

肥皂泡2 发布于 2013/05/03 10:39
阅读 1K+
收藏 0

 问个问题:单台机器做hadoop伪分布式,每十分钟往里面存一个seq文件,df.replication=1,当程序运行几个小时或更久后,为什么会丢失很多块? 管理页面上显示:WARNING : There are about 98 missing blocks. Please check the log or run fsck 

这是什么原因啊!!!

以下是问题补充:

@肥皂泡2:datanode Log: 2013-05-04 19:10:16,684 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: First Verification failed for blk_-3099835012436947671_3601. Exception : java.io.IOException: Block blk_-3099835012436947671_3601 is not valid. at org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072) at org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035) at org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045) at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94) at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81) at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453) at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519) at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617) at java.lang.Thread.run(Thread.java:662) 2013-05-04 19:10:16,686 INFO org.mortbay.log: Completed FSVolumeSet.checkDirs. Removed=0volumes. List of current volumes: /letv/fet/live/data/hadoop/hdfs/data/current 2013-05-04 19:10:16,686 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second Verification failed for blk_-3099835012436947671_3601. Exception : java.io.IOException: Block blk_-3099835012436947671_3601 is not valid. at org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072) at org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035) at org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045) at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94) at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81) at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453) at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519) at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617) at java.lang.Thread.run(Thread.java:662) 2013-05-04 19:10:16,686 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Reporting bad block blk_-3099835012436947671_3601 to namenode. (2013/05/04 19:27)
加载中
0
lateron
lateron
看看日志
肥皂泡2
肥皂泡2
没怎么看明白日志 这种情况大概是那方面出问题了啊!!!
返回顶部
顶部