Could Only Be Replicated To …

Could Only Be Replicated To …

YDB中出现如下报错的时候,需要调整HADOOP的相关参数

LuncenIndexMergerWithTimeout
19_20170723_124216:170723124200000277 org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /data/ycloud/ydb/ydbpath/index/fhbcp/index/y_100_2000_1718/part-00015/ydb_20170723124216_120034_3414772784_realtime/_0.si could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and no node(s) are excluded in this operation.; at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1641); at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3198); at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3122); at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:843); at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:500); at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocol

此时的GC状态-OLD区域基本已经满了

 

==调整datanodeNamenode的内存为5G以上

==调整YDBexecuter的内存为16G以上

==默认是4096调整为48000

<property>
<name>dfs.datanode.max.transfer.threads</name>
<value>4096</value>
<description>
Specifies the maximum number of threads to use for transferring data
in and out of the DN.
</description>
</property>

==默认是10 调整为300

<property>
<name>dfs.datanode.handler.count</name>
<value>10</value>
<description>The number of server threads for the datanode.</description>
</property>

==默认是10 调整为600

<property>
<name>dfs.namenode.handler.count</name>
<value>10</value>
<description>The number of server threads for the namenode.</description>
</property>