Unable to close file because the last block does not have enough number of repli

Unable to close file because the last block does not have enough number of replicas.

LuncenIndexMergerWithTimeout
1_20170826_231709:170826231700000001 java.io.IOException: Unable to close file because the last block does not have enough number of replicas.;java.io.IOException: Unable to close file because the last block does not have enough number of replicas.; at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2306); at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2267); at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2232); at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72); at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106); at java.io.FilterOutputStream.close(FilterOutputStream.java:159); at java.io.FilterOutputStream.close(FilterOutputStream.java:159); at cn.net.ydbmix.svn2.ydb.hdfsBlockDirectore.HdfsFileWriter.close(HdfsFileWriter.java:83); at cn.net.ydbmix.svn2.ydb.hdfsBlockDirectore.BlockCacheRamOutput.close(BlockCacheRamOutput.java:46);lucene.index.SegmentInfos.write(SegmentInfos.java:579);lucene.index.SegmentInf
mergerexcetion
9_20170826_231728:170826231700000002 java.util.concurrent.ExecutionException: java.lang.Exception: java.io.IOException: Unable to close file because the last block does not have enough number of replicas.;java.util.concurrent.ExecutionException: java.lang.Exception: java.io.IOException: Unable to close file because the last block does not have enough number of replicas.; at cn.net.ydbmix.svn2.ydb.ydbDirectory.realtime.LuceneIndexMergerWithTimeout.merger(LuceneIndexMergerWithTimeout.java:509); at cn.net.ydbmix.svn2.ydb.ydbDirectory.realtime.RealTimeDirectorUtils.merger(RealTimeDirectorUtils.java:113); at cn.net.ydbmix.svn2.ydb.ydbDirectory.realtime.RealTimeDirectoryData.commit2Merger2disk(RealTimeDirectoryData.java:1220); at cn.net.ydbmix.svn2.ydb.ydbDirectory.realtime.RealTimeDirectoryData.ramMerger2disk(RealTimeDirectoryData.java:1285); at cn.net.ydbmix.svn2.ydb.ydbDirectory.RealTimeDirectory.commitBuffer2Search(RealTimeDirectory.java:1731); at cn.net.ydbmix.svn2.ydb.ydbDirectory.RealTimeDirectory.flushDocList(RealTimeDirectory.java:1690); at c

 

 

增加locateFollowingBlock办法的重试次数

示例:

Exception in thread "main"java.io.IOException: Unable to close file because the last block does not haveenough number of replicas.

分析:

可能是NameNode过于繁忙,locateFollowingBlock方法请求Name Node为文件添加新块发生错误,无法定位下一个块。建议增加locateFollowingBlock方法重试次数参

解决办法:

修改默认作业参数dfs.client.block.write.locateFollowingBlock.retries15 (默认是5)