1
回答
elasticsearch 索引分片问题
百度AI开发者大赛带你边学边开发,赢100万奖金,加群:418589053   

es集群,机器三台,node1上的两个分片只能启动一个,另外一个分片一直处于:1 unassigned shards

前期一直没有出现问题,一直到磁盘空间占用达到75%出现问题。

从head插件上看到的报错信息如下:

{

    "state": "UNASSIGNED",
    "primary": true,
    "node": null,
    "relocating_node": null,
    "shard": 1,
    "index": "originaldata_index",
    "version": 6,
    "unassigned_info": {
        "reason": "ALLOCATION_FAILED",
        "at": "2018-05-17T21:34:39.295Z",
        "details": "failed recovery, failure IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: CorruptIndexException[i_o_exception: failed engine (reason: [corrupt file (source: [index])]) (resource=preexisting_corruption)]; nested: NotSerializableExceptionWrapper[i_o_exception: failed engine (reason: [corrupt file (source: [index])])]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))]; "
    }

}

找不到问题所在,请大牛指教。

以下为错误日志(请忽略主机路由的错误):

[2018-05-17 15:05:20,959][ERROR][index.engine             ] [node-1] [originaldata_index][1] failed to merge
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))
	at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:334)
	at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:451)
	at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:333)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:317)
	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4075)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
	at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
[2018-05-17 15:05:20,959][DEBUG][action.bulk              ] [node-1] failed to execute [BulkShardRequest to [originaldata_index] containing [95] requests] on [[originaldata_index][1]]
[originaldata_index][[originaldata_index][1]] EngineClosedException[CurrentState[CLOSED] Closed]
	at org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:329)
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:451)
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:601)
	at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:836)
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:237)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 15:05:21,068][DEBUG][action.bulk              ] [node-1] failed to execute [BulkShardRequest to [originaldata_index] containing [96] requests] on [[originaldata_index][1]]
[originaldata_index][[originaldata_index][1]] EngineClosedException[CurrentState[CLOSED] Closed]
	at org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:329)
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:451)
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:601)
	at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:836)
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:237)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 15:05:21,086][DEBUG][action.bulk              ] [node-1] failed to execute [BulkShardRequest to [originaldata_index] containing [97] requests] on [[originaldata_index][1]]
[originaldata_index][[originaldata_index][1]] EngineClosedException[CurrentState[CLOSED] Closed]
	at org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:329)
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:451)
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:601)
	at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:836)
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:237)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 15:05:21,251][WARN ][index.engine             ] [node-1] [originaldata_index][1] failed engine [corrupt file (source: [index])]
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
	at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:530)
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:457)
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:601)
	at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:836)
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:237)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))
	at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:334)
	at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:451)
	at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:333)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:317)
	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4075)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
	at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
[2018-05-17 15:05:21,265][DEBUG][action.bulk              ] [node-1] [originaldata_index][1] failed to execute bulk item (index) index {[originaldata_index][originaldata][932d795846f530037e74475dd597dd1c], source[{"id":"932d795846f530037e74475dd597dd1c","phone":"18722084153\n"}]}
[originaldata_index][[originaldata_index][1]] IndexFailedEngineException[Index failed for [originaldata#932d795846f530037e74475dd597dd1c]]; nested: AlreadyClosedException[this IndexWriter is closed]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))];
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:462)
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:601)
	at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:836)
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:237)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
	at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:530)
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:457)
	... 16 more
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))
	at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:334)
	at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:451)
	at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:333)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:317)
	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4075)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
	at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
[2018-05-17 15:05:21,265][WARN ][indices.cluster          ] [node-1] [[originaldata_index][1]] marking and sending shard failed due to [engine failure, reason [corrupt file (source: [index])]]
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
	at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:530)
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:457)
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:601)
	at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:836)
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:237)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))
	at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:334)
	at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:451)
	at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:333)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:317)
	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4075)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
	at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
[2018-05-17 15:05:21,266][DEBUG][action.bulk              ] [node-1] failed to execute [BulkShardRequest to [originaldata_index] containing [94] requests] on [[originaldata_index][1]]
[originaldata_index][[originaldata_index][1]] IllegalIndexShardStateException[CurrentState[CLOSED] operation only allowed when started/recovering, origin [PRIMARY]]
	at org.elasticsearch.index.shard.IndexShard.ensureWriteAllowed(IndexShard.java:1068)
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:593)
	at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:836)
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:237)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 15:05:21,266][WARN ][cluster.action.shard     ] [node-1] [originaldata_index][1] received shard failed for target shard [[originaldata_index][1], node[rl69tZt6R7Cq3YxD551NGQ], [P], v[6], s[STARTED], a[id=n1brrHERTZ2yGmdLmZDFyA]], indexUUID [GrioJA7rSEW9WrKqr9Ukdw], message [engine failure, reason [corrupt file (source: [index])]], failure [AlreadyClosedException[this IndexWriter is closed]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))]; ]
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
	at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:530)
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:457)
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:601)
	at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:836)
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:237)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))
	at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:334)
	at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:451)
	at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:333)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:317)
	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4075)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
	at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
[2018-05-17 15:05:21,269][INFO ][cluster.routing.allocation] [node-1] Cluster health status changed from [GREEN] to [RED] (reason: [shards failed [[originaldata_index][1]] ...]).
[2018-05-17 15:29:47,730][INFO ][cluster.service          ] [node-1] removed {{node-3}{vb5-mZf4TWCtnxfDUNnEIQ}{192.168.31.36}{192.168.31.36:9300},}, reason: zen-disco-node_failed({node-3}{vb5-mZf4TWCtnxfDUNnEIQ}{192.168.31.36}{192.168.31.36:9300}), reason transport disconnected
[2018-05-17 15:30:10,523][INFO ][cluster.service          ] [node-1] added {{node-3}{zcKVnP3XRaG9s-VXzoXgdw}{192.168.31.36}{192.168.31.36:9300},}, reason: zen-disco-join(join from node[{node-3}{zcKVnP3XRaG9s-VXzoXgdw}{192.168.31.36}{192.168.31.36:9300}])
[2018-05-17 15:30:50,259][WARN ][bootstrap                ] running as ROOT user. this is a bad idea!
[2018-05-17 15:30:50,270][WARN ][bootstrap                ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2018-05-17 15:30:50,954][INFO ][node                     ] [node-1] version[2.3.2], pid[31008], build[b9e4a6a/2016-04-21T16:03:47Z]
[2018-05-17 15:30:50,955][INFO ][node                     ] [node-1] initializing ...
[2018-05-17 15:30:51,325][INFO ][plugins                  ] [node-1] modules [reindex, lang-expression, lang-groovy], plugins [head, kopf], sites [kopf, head]
[2018-05-17 15:30:51,343][INFO ][env                      ] [node-1] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [55.1gb], net total_space [232.5gb], spins? [no], types [ext4]
[2018-05-17 15:30:51,343][INFO ][env                      ] [node-1] heap size [5.9gb], compressed ordinary object pointers [true]
[2018-05-17 15:30:51,343][WARN ][env                      ] [node-1] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2018-05-17 15:30:52,607][INFO ][node                     ] [node-1] initialized
[2018-05-17 15:30:52,607][INFO ][node                     ] [node-1] starting ...
[2018-05-17 15:30:52,664][INFO ][transport                ] [node-1] publish_address {192.168.31.34:9300}, bound_addresses {[::]:9300}
[2018-05-17 15:30:52,667][INFO ][discovery                ] [node-1] myes/8G3EZ8lpT8a5V07jx0kVSg
[2018-05-17 15:30:55,713][INFO ][cluster.service          ] [node-1] detected_master {node-2}{SfDHU5VFTTqELS4box-Lww}{192.168.31.35}{192.168.31.35:9300}, added {{node-3}{zcKVnP3XRaG9s-VXzoXgdw}{192.168.31.36}{192.168.31.36:9300},{node-2}{SfDHU5VFTTqELS4box-Lww}{192.168.31.35}{192.168.31.35:9300},}, reason: zen-disco-receive(from master [{node-2}{SfDHU5VFTTqELS4box-Lww}{192.168.31.35}{192.168.31.35:9300}])
[2018-05-17 15:30:55,798][INFO ][http                     ] [node-1] publish_address {192.168.31.34:9200}, bound_addresses {[::]:9200}
[2018-05-17 15:30:55,798][INFO ][node                     ] [node-1] started
[2018-05-17 16:07:23,987][WARN ][bootstrap                ] running as ROOT user. this is a bad idea!
[2018-05-17 16:07:23,997][WARN ][bootstrap                ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2018-05-17 16:07:24,676][INFO ][node                     ] [node-1] version[2.3.2], pid[31340], build[b9e4a6a/2016-04-21T16:03:47Z]
[2018-05-17 16:07:24,676][INFO ][node                     ] [node-1] initializing ...
[2018-05-17 16:07:25,040][INFO ][plugins                  ] [node-1] modules [reindex, lang-expression, lang-groovy], plugins [head, kopf], sites [kopf, head]
[2018-05-17 16:07:25,058][INFO ][env                      ] [node-1] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [55.2gb], net total_space [232.5gb], spins? [no], types [ext4]
[2018-05-17 16:07:25,058][INFO ][env                      ] [node-1] heap size [5.9gb], compressed ordinary object pointers [true]
[2018-05-17 16:07:25,059][WARN ][env                      ] [node-1] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2018-05-17 16:07:26,300][INFO ][node                     ] [node-1] initialized
[2018-05-17 16:07:26,300][INFO ][node                     ] [node-1] starting ...
[2018-05-17 16:07:26,362][INFO ][transport                ] [node-1] publish_address {192.168.31.34:9300}, bound_addresses {[::]:9300}
[2018-05-17 16:07:26,365][INFO ][discovery                ] [node-1] myes/AOyyCelZT-mm9ExiA98kjw
[2018-05-17 16:07:29,418][INFO ][cluster.service          ] [node-1] detected_master {node-2}{SfDHU5VFTTqELS4box-Lww}{192.168.31.35}{192.168.31.35:9300}, added {{node-2}{SfDHU5VFTTqELS4box-Lww}{192.168.31.35}{192.168.31.35:9300},{node-3}{zcKVnP3XRaG9s-VXzoXgdw}{192.168.31.36}{192.168.31.36:9300},}, reason: zen-disco-receive(from master [{node-2}{SfDHU5VFTTqELS4box-Lww}{192.168.31.35}{192.168.31.35:9300}])
[2018-05-17 16:07:29,513][INFO ][http                     ] [node-1] publish_address {192.168.31.34:9200}, bound_addresses {[::]:9200}
[2018-05-17 16:07:29,513][INFO ][node                     ] [node-1] started
[2018-05-17 16:24:52,628][INFO ][node                     ] [node-1] stopping ...
[2018-05-17 16:24:53,684][INFO ][node                     ] [node-1] stopped
[2018-05-17 16:24:53,685][INFO ][node                     ] [node-1] closing ...
[2018-05-17 16:24:53,687][INFO ][node                     ] [node-1] closed
[2018-05-17 17:58:33,769][WARN ][bootstrap                ] running as ROOT user. this is a bad idea!
[2018-05-17 17:58:33,780][WARN ][bootstrap                ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2018-05-17 17:58:34,859][INFO ][node                     ] [node-1] version[2.3.2], pid[3379], build[b9e4a6a/2016-04-21T16:03:47Z]
[2018-05-17 17:58:34,859][INFO ][node                     ] [node-1] initializing ...
[2018-05-17 17:58:35,501][INFO ][plugins                  ] [node-1] modules [reindex, lang-expression, lang-groovy], plugins [head, kopf], sites [head, kopf]
[2018-05-17 17:58:35,552][INFO ][env                      ] [node-1] using [1] data paths, mounts [[/sda (/dev/sda)]], net usable_space [271.6gb], net total_space [440.1gb], spins? [no], types [ext3]
[2018-05-17 17:58:35,552][INFO ][env                      ] [node-1] heap size [5.9gb], compressed ordinary object pointers [true]
[2018-05-17 17:58:35,552][WARN ][env                      ] [node-1] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2018-05-17 17:58:37,116][INFO ][node                     ] [node-1] initialized
[2018-05-17 17:58:37,116][INFO ][node                     ] [node-1] starting ...
[2018-05-17 17:58:37,170][INFO ][transport                ] [node-1] publish_address {192.168.31.34:9300}, bound_addresses {[::]:9300}
[2018-05-17 17:58:37,173][INFO ][discovery                ] [node-1] myes/WpB_MWLhSL6AqxZ8s7Fzew
[2018-05-17 17:58:37,230][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x1a3f473a]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:58:38,684][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x74346bf5]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:58:40,188][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xf06c3b53]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:58:40,202][INFO ][cluster.service          ] [node-1] new_master {node-1}{WpB_MWLhSL6AqxZ8s7Fzew}{192.168.31.34}{192.168.31.34:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2018-05-17 17:58:40,211][INFO ][http                     ] [node-1] publish_address {192.168.31.34:9200}, bound_addresses {[::]:9200}
[2018-05-17 17:58:40,211][INFO ][node                     ] [node-1] started
[2018-05-17 17:58:40,245][INFO ][gateway                  ] [node-1] recovered [1] indices into cluster_state
[2018-05-17 17:58:48,820][INFO ][cluster.service          ] [node-1] added {{node-2}{qBwKtspLSnieyd1WxoTBfQ}{192.168.31.35}{192.168.31.35:9300},}, reason: zen-disco-join(join from node[{node-2}{qBwKtspLSnieyd1WxoTBfQ}{192.168.31.35}{192.168.31.35:9300}])
[2018-05-17 17:58:57,631][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x1dbbbc2c]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:58:57,632][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x77b11823]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:00,670][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xf39d262c]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:00,672][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x9a3b8274]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:00,673][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xb7d15bee]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:03,701][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xe402e71e]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:03,702][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x73a9c3b0]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:06,719][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x9ec4c7d1]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:06,719][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x8c16d139]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:06,720][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x9ea278e0]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:09,731][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x420e6039]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:09,731][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xd4812ab2]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:12,745][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x1b1a123d]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:12,745][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xa2e71c03]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:15,758][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x8a219807]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:15,759][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xe86171b2]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:15,760][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xf0565cc4]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:18,774][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x4310f239]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:18,775][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x5c9c32b5]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:18,775][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x5a071143]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:21,787][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xbbc70236]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:21,788][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x149cb3e4]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:21,789][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x9e0e4498]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:24,804][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xace40207]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:24,805][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xe2d96c36]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:27,820][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xf369a5b2]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:27,821][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x6df06fab]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:30,836][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xb1e99315]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:30,838][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xd1e1ba52]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:33,850][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x606f8946]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:33,851][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xc526ad61]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:36,861][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x726fab60]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:36,862][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xbd57c816]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:39,876][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x7fcfac83]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:39,877][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0x040b7b30]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:42,890][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xeb0ab4b0]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:42,891][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xb54ff1d9]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:45,908][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xa7ffb2f8]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:45,909][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xd390cd3f]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:48,922][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xbbb98be2]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:48,923][WARN ][transport.netty          ] [node-1] exception caught on transport layer [[id: 0xe5b91976]], closing connection
java.net.NoRouteToHostException: 没有到主机的路由
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-05-17 17:59:51,971][INFO ][cluster.service          ] [node-1] added {{node-3}{2TOYtEztQYifnp-f-4hxDg}{192.168.31.36}{192.168.31.36:9300},}, reason: zen-disco-join(join from node[{node-3}{2TOYtEztQYifnp-f-4hxDg}{192.168.31.36}{192.168.31.36:9300}])
[2018-05-17 18:06:23,128][WARN ][indices.cluster          ] [node-1] [[originaldata_index][1]] marking and sending shard failed due to [failed recovery]
[originaldata_index][[originaldata_index][1]] IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: CorruptIndexException[i_o_exception: failed engine (reason: [corrupt file (source: [index])]) (resource=preexisting_corruption)]; nested: NotSerializableExceptionWrapper[i_o_exception: failed engine (reason: [corrupt file (source: [index])])]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))];
	at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:224)
	at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
	at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.lucene.index.CorruptIndexException: i_o_exception: failed engine (reason: [corrupt file (source: [index])]) (resource=preexisting_corruption)
	at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:583)
	at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:563)
	at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:197)
	... 5 more
Caused by: NotSerializableExceptionWrapper[i_o_exception: failed engine (reason: [corrupt file (source: [index])])]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))];
	at org.elasticsearch.index.engine.Engine.failEngine(Engine.java:555)
	at org.elasticsearch.index.engine.Engine.maybeFailEngine(Engine.java:576)
	at org.elasticsearch.index.engine.InternalEngine.maybeFailEngine(InternalEngine.java:907)
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:461)
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:601)
	at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:836)
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:237)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	... 3 more
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))
	at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:334)
	at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:451)
	at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:333)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:317)
	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4075)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
	at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
[2018-05-17 18:06:23,130][WARN ][cluster.action.shard     ] [node-1] [originaldata_index][1] received shard failed for target shard [[originaldata_index][1], node[WpB_MWLhSL6AqxZ8s7Fzew], [P], v[1], s[INITIALIZING], a[id=e2TuDlPTTuuvQCDa5bXRzw], unassigned_info[[reason=INDEX_CREATED], at[2018-05-17T10:06:23.097Z], details[force allocation from previous reason CLUSTER_RECOVERED, null]]], indexUUID [GrioJA7rSEW9WrKqr9Ukdw], message [failed recovery], failure [IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: CorruptIndexException[i_o_exception: failed engine (reason: [corrupt file (source: [index])]) (resource=preexisting_corruption)]; nested: NotSerializableExceptionWrapper[i_o_exception: failed engine (reason: [corrupt file (source: [index])])]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))]; ]
[originaldata_index][[originaldata_index][1]] IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: CorruptIndexException[i_o_exception: failed engine (reason: [corrupt file (source: [index])]) (resource=preexisting_corruption)]; nested: NotSerializableExceptionWrapper[i_o_exception: failed engine (reason: [corrupt file (source: [index])])]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))];
	at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:224)
	at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
	at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.lucene.index.CorruptIndexException: i_o_exception: failed engine (reason: [corrupt file (source: [index])]) (resource=preexisting_corruption)
	at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:583)
	at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:563)
	at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:197)
	... 5 more
Caused by: NotSerializableExceptionWrapper[i_o_exception: failed engine (reason: [corrupt file (source: [index])])]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))];
	at org.elasticsearch.index.engine.Engine.failEngine(Engine.java:555)
	at org.elasticsearch.index.engine.Engine.maybeFailEngine(Engine.java:576)
	at org.elasticsearch.index.engine.InternalEngine.maybeFailEngine(InternalEngine.java:907)
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:461)
	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:601)
	at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:836)
	at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:237)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	... 3 more
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=f7ff1d4a actual=2a0a27c7 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/local/elasticsearch1/data/myes/nodes/0/indices/originaldata_index/1/index/_fses.cfs") [slice=_fses_Lucene50_0.tim]))
	at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:334)
	at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:451)
	at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:333)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:317)
	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4075)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
	at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)

 

 

举报
顶部