{"id":667,"date":"2022-02-16T15:47:33","date_gmt":"2022-02-16T07:47:33","guid":{"rendered":"https:\/\/www.luyouli.com\/?p=667"},"modified":"2022-03-04T16:45:06","modified_gmt":"2022-03-04T08:45:06","slug":"%e5%a4%84%e7%90%86zookeeper%e8%b6%85%e6%97%b6%e5%af%bc%e8%87%b4hregionserver%e5%ae%95%e6%9c%ba","status":"publish","type":"post","link":"https:\/\/www.luyouli.com\/?p=667","title":{"rendered":"\u5904\u7406HDFS DataXceiver error processing WRITE_BLOCK operation\u5bfc\u81f4Hregionserver\u5b95\u673a\u53ca\u5904\u7406\u8fc7\u7a0b"},"content":{"rendered":"\n

\u6700\u8fd1\u65b0\u642d\u5efa\u4e86\u4e00\u5957HBase\uff082.3.x\uff09+Hadoop\uff083.2.x\uff09\u9ad8\u7248\u672c\u6d4b\u8bd5\u73af\u5883<\/p>\n\n\n\n

\u6bcf\u5230\u534a\u591c\u7684\u65f6\u5019\u5c31\u4f1a\u5f00\u59cb\u5b95\u8282\u70b9\uff0c\u7136\u540e\u7ea7\u8054\uff0c\u7136\u540e\u96ea\u5d29\uff0c\u7136\u540e\u7b2c\u4e8c\u5929\u5c31\u88abDISS\u00b7\u00b7\u00b7<\/p>\n\n\n\n

\u5f7b\u67e5\u4e86\u4e0b\u65e5\u5fd7\uff0c\u53d1\u73b0\u6709\u51e0\u4e2a\u95ee\u9898\u70b9\uff1a<\/p>\n\n\n\n

1\u3001ERROR [regionserver\/hbase1:16020<\/em>] zookeeper.ZKWatcher<\/em>: regionserver:16020<\/em>-0xxxxxx, quorum=master1:2181<\/em>,master2:2181<\/em>,master3:2181<\/em>, baseZNode=\/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for \/hbase\/master<\/p>\n\n\n\n

2\u3001INFO [main-SendThread(master2:2181<\/em>)] zookeeper.ClientCnxn<\/em>: Session establishment complete on server master2\/172<\/em>.xx<\/em>x.xxx<\/em>.1:2181<\/em>, sessionid = 0xxxxxx, negotiated timeout = 30000<\/em><\/p>\n\n\n\n

3\u3001INFO [JvmPauseMonitor] util.JvmPauseMonitor<\/em>: Detected pause in JVM or host machine (eg GC): pause of approximately 3213ms
GC pool ‘ParNew’ had collection(s): count=1<\/em> time=48ms
GC pool ‘ConcurrentMarkSweep’ had collection(s): count=1<\/em> time=3608ms<\/p>\n\n\n\n

4\u3001INFO [AsyncFSWAL-0<\/em>–hdfs:\/\/master\/hbase<\/em>] wal.AbstractFSWAL<\/em>: Slow sync cost: 321<\/em> ms, current pipeline: [DatanodeInfoWithStorage[172<\/em>.xxx<\/em>.xxx<\/em>.1:9866<\/em>,DS-xxx<\/em>,DISK], DatanodeInfoWithStorage[172<\/em>.xxx<\/em>.xxx<\/em>.2:9866<\/em>,DS-xxx,DISK], DatanodeInfoWithStorage[172<\/em>.xxx<\/em>.xxx<\/em>.3:9866<\/em>,DS-xxx<\/em>,DISK]]
INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache<\/em>: totalSize=9<\/em>.13<\/em> GB, freeSize=454<\/em>.38<\/em> MB, max=9<\/em>.57<\/em> GB, blockCount=142290<\/em>, accesses=26146057<\/em>, hits=22935485<\/em>, hitRatio=87<\/em>.72<\/em>%, , cachingAccesses=25568452<\/em>, cachingHits=22876827<\/em>, cachingHitsRatio=89<\/em>.47<\/em>%, evictions=2841<\/em>, evicted=2481248<\/em>, evictedPerRun=873<\/em>.3713481168603
<\/em>INFO [hbase2:16020Replication Statistics #0<\/em>] regionserver.Replication<\/em>: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B<\/em><\/p>\n\n\n\n

\u8fd8\u6709\u4e00\u4e2azookeeper leader\u65e5\u5fd7<\/p>\n\n\n\n

5\u3001WARN [NIOWorkerThread-2:NIOServerCnxn@364] – Unexpected exception EndOfStreamException: Unable to read additional data from client, it probably closed the socket: address = \/172.xxx.xxx.7:39505, session = 0xxxxx at org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:163) at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:326) at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522) at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)<\/p>\n\n\n\n

\u6839\u636e\u4e0a\u9762\u8fd9\u4e9b\u65e5\u5fd7\u7684\u70b9\uff0c\u7efc\u5408\u8003\u91cf\u4e4b\u540e\uff0c\u5224\u65ad\u662fWAL\u540c\u6b65\u7684\u65f6\u5019\u94fe\u63a5\u7531\u4e8e\u6570\u636e\u91cf\u592a\u5927\u5bfc\u81f4\u8d85\u8fc7zookeeper\u7684tickTime\u4e86\uff0c\u76f4\u63a5\u628a\u6570\u503c\u63d0\u5347\u4e3a3\u5206\u949f\u8bd5\u8bd5<\/p>\n\n\n\n

1\u3001\u589e\u52a0\/\u4fee\u6539hbase-site.xml<\/p>\n\n\n\n

<property>
<name>zookeeper.session.timeout<\/name>
<value>180000<\/value>
<\/property><\/p>\n\n\n\n

2\u3001\u589e\u52a0\/\u4fee\u6539zoo.cfg<\/p>\n\n\n\n

tickTime=9000
minSessionTimeout=18000
maxSessionTimeout=180000<\/p>\n\n\n\n

\u7528ansible\u66f4\u65b0\u5b8c\u914d\u7f6e\u540e\uff0c\u76f4\u63a5\u91cd\u542fzookeeper\u96c6\u7fa4\u548cHBase\u96c6\u7fa4<\/p>\n\n\n\n

————————–\u5206\u5272\u7ebf————————————<\/p>\n\n\n\n

2022\u5e742\u670821\u65e5\u66f4\u65b0\uff1a<\/p>\n\n\n\n

\u7136\u540e\u53c8\u8dd1\u4e86\u4e24\u5929\u53d1\u73b0\u8fd8\u662f\u4f1a\u5b95\u673a\uff0c\u4f46\u662f\u9891\u7387\u6ca1\u6709\u90a3\u4e48\u9ad8\u4e86\uff0c\u800c\u4e14\u5728\u5b95\u673a\u4e4b\u524d\u5185\u5b58\u57fa\u672c\u4e0a\u90fd\u662f98%\u4ee5\u4e0a\uff0c\u8bf4\u660e\u8fd8\u6709\u5730\u65b9\u9700\u8981\u4fee\u6539\uff0c\u7136\u540e\u5bf9\u6574\u4e2a\u96c6\u7fa4\u7684\u6240\u6709\u65e5\u5fd7\u5168\u90e8\u8fc7\u4e86\u4e00\u904d\u53d1\u73b0\u4e86\u4e4b\u524d\u6ca1\u6709\u53d1\u73b0\u7684\u95ee\u9898\uff1a<\/p>\n\n\n\n

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: hbase01:9866:DataXceiver error processing WRITE_BLOCK operation src: \/172.xxx.xxx.1:52236 dst: \/172.xxx.xxx.2:9866
java.io.IOException: Premature EOF from inputStream<\/p>\n\n\n\n

\u6587\u4ef6\u64cd\u4f5c\u8d85\u79df\u671f\uff0c\u5b9e\u9645\u4e0a\u5c31\u662fdata stream\u64cd\u4f5c\u8fc7\u7a0b\u4e2d\u6587\u4ef6\u88ab\u5220\u9664\u4e86\uff0c\u9700\u8981\u8c03\u6574dfs.datanode.max.transfer.threads\u53c2\u6570\u7684\u4e0a\u9650<\/p>\n\n\n\n

\u68c0\u67e5\u4e0b\u4e86\u914d\u7f6e\u6587\u4ef6\uff0c\u53d1\u73b0\u914d\u7f6e\u7684\u662f4096\uff0c\u76f4\u63a5\u7ffb\u4e00\u500d\u603c\u4e0a\u53bb<\/p>\n\n\n\n

<property>\n    <name>dfs.datanode.max.transfer.threads<\/name>\n    <value>8192<\/value>\n<\/property><\/pre>\n\n\n\n
\u9488\u5bf9WAL\u7684\u914d\u7f6e\u4e5f\u8fdb\u884c\u4fee\u6539\uff0c\u9488\u5bf9\u6211\u4eec\u8bfb\u591a\u5199\u5c11\u7684\u4e1a\u52a1\u4fee\u6539\u4e86\u8bfb\u5199\u7f13\u5b58<\/p>\n\n\n\n
  <property>\n    <name>hfile.block.cache.size<\/name>\n    <value>0.3<\/value>\n  <\/property>\n\n  <property>\n    <name>hbase.regionserver.global.memstore.size<\/name>\n    <value>0.2<\/value>\n  <\/property><\/code><\/pre>\n\n\n\n
\u7528ansible\u66f4\u65b0\u5b8c\u914d\u7f6e\u540e\uff0c\u76f4\u63a5\u91cd\u542fHBase\u96c6\u7fa4\u548cHadoop\u96c6\u7fa4<\/p>\n\n\n\n
—————————\u5206\u5272\u7ebf————————————<\/p>\n\n\n\n
2022\u5e742\u670824\u65e5\u66f4\u65b0\uff1a<\/p>\n\n\n\n
\u4eca\u5929\u53c8\u96ea\u5d29\u4e86\uff0c\u6211\u5c31X\u4e86\uff0c\u904d\u5386\u4e86\u6240\u6709\u8282\u70b9\u7684\u65e5\u5fd7\u548c\u914d\u7f6e\u6587\u4ef6\uff0c\u7ec8\u4e8e\u53d1\u73b0\u95ee\u9898\u6240\u5728<\/p>\n\n\n\n
\u6211\u66f4\u65b0\u4e86hadoop\u7684hdfs-site.xml\u6587\u4ef6<\/p>\n\n\n\n
\u4f46\u662f<\/p>\n\n\n\n
\u4f46\u662f<\/p>\n\n\n\n
\u4f46\u662f<\/p>\n\n\n\n
\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7<\/p>\n\n\n\n
\u6211\u6ca1\u6709\u628ahdfs-site.xml\u6587\u4ef6\u66f4\u65b0\u5230hbase\u7684conf\u91cc\u9762<\/p>\n\n\n\n
\u6211\u6ca1\u6709\u628ahdfs-site.xml\u6587\u4ef6\u66f4\u65b0\u5230hbase\u7684conf\u91cc\u9762<\/p>\n\n\n\n
\u6211\u6ca1\u6709\u628ahdfs-site.xml\u6587\u4ef6\u66f4\u65b0\u5230hbase\u7684conf\u91cc\u9762<\/p>\n\n\n\n
\u6211\u5c6e\u8278\u8294\u833b<\/p>\n\n\n\n
—————————\u5206\u5272\u7ebf————————————<\/p>\n\n\n\n
2022\u5e742\u670826\u65e5\u66f4\u65b0\uff1a<\/p>\n\n\n\n
\u4eca\u5929\u00b7\u00b7\u00b7\u53c8\u5b95\u4e86\u00b7\u00b7\u00b7\u6211\u5df2\u7ecf\u65e0F**K\u8bf4\u4e86\uff0c\u6211\u5df2\u7ecf\u5f00\u59cb\u6000\u7591\u662f\u4e0d\u662f\u8fd9\u4e2a2.3.x\u7248\u672c\u548cCMS\u7684\u8054\u5408BUG\u4e86<\/p>\n\n\n\n
\u5728\u9ad8\u4eba\u6307\u70b9\u4e0b\uff0c\u6539\u4e86\u4e0bnofile\u3001sigpending\u548cnproc<\/p>\n\n\n\n
user soft sigpending 655350
user hard sigpending 655350
user soft nofile 655360
user hard nofile 655360
user soft nproc 655360
user hard nproc 655360<\/p>\n\n\n\n
\u91cd\u542f\u00b7\u00b7\u00b7\u9759\u5019\u00b7\u00b7\u4f73\u97f3\u00b7\u00b7<\/p>\n\n\n\n
—————————\u5206\u5272\u7ebf————————————<\/p>\n\n\n\n
2022\u5e742\u670827\u65e5\u66f4\u65b0\uff1a<\/p>\n\n\n\n
\u597d\u5427\uff0c\u4eca\u5929\u53c8\u5b95\u6389\u4e86\uff0c\u767e\u601d\u4e0d\u5f97\u5176\u89e3\u554a\u767e\u601d\u4e0d\u5f97\u5176\u89e3<\/p>\n\n\n\n
\u7ec8\u6781\u6740\u62db\uff1a\u964d\u7ea7<\/p>\n\n\n\n
\u628ahbase\u7684\u7248\u672c\u4ece2.3.7\u964d\u7ea7\u52302.3.6<\/p>\n\n\n\n
\u7b49\u00b7\u00b7\u00b7\u7ed3\u679c<\/p>\n\n\n\n
—————————\u5206\u5272\u7ebf————————————<\/p>\n\n\n\n
2022\u5e743\u67082\u65e5\u66f4\u65b0\uff1a<\/p>\n\n\n\n
\u5728hbase-env.sh\u91cc\u628aG1GC\u914d\u7f6e\u4e0a\u4e86<\/p>\n\n\n\n
export HBASE_OPTS=\"-XX:+UseG1GC\"
export HBASE_MASTER_OPTS=\"$HBASE_MASTER_OPTS -XX:+UseG1GC -Xmx12g -Xms12g -XX:+UnlockExperimentalVMOptions -XX:-ResizePLAB -XX:MaxGCPauseMillis=100 -XX:ParallelGCThreads=4 -XX:ConcGCThreads=1 -XX:G1HeapWastePercent=10 -XX:InitiatingHeapOccupancyPercent=45 -XX:G1MixedGCLiveThresholdPercent=80 -XX:G1NewSizePercent=1 -XX:G1MaxNewSizePercent=15\"
export HBASE_REGIONSERVER_OPTS=\"$HBASE_REGIONSERVER_OPTS -XX:+UseG1GC -Xmx28g -Xms28g -XX:+UnlockExperimentalVMOptions -XX:-ResizePLAB -XX:MaxGCPauseMillis=100 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=2 -XX:G1HeapWastePercent=10 -XX:InitiatingHeapOccupancyPercent=45 -XX:G1MixedGCLiveThresholdPercent=80 -XX:G1NewSizePercent=1 -XX:G1MaxNewSizePercent=15\"<\/pre>\n\n\n\n
\u8fd9\u6b21\uff0c\u6ca1\u95ee\u9898\u4e86<\/p>\n","protected":false},"excerpt":{"rendered":"
\u6700\u8fd1\u65b0\u642d\u5efa\u4e86\u4e00\u5957HBase\uff082.3.x\uff09+Hadoop\uff083.2.x\uff09\u9ad8\u7248\u672c\u6d4b\u8bd5\u73af\u5883 \u6bcf\u5230\u534a\u591c\u7684\u65f6\u5019\u5c31\u4f1a\u5f00\u59cb\u5b95\u8282 […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[23],"tags":[],"class_list":["post-667","post","type-post","status-publish","format-standard","hentry","category-hbase"],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":6}},"views":529,"_links":{"self":[{"href":"https:\/\/www.luyouli.com\/index.php?rest_route=\/wp\/v2\/posts\/667","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.luyouli.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.luyouli.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.luyouli.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.luyouli.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=667"}],"version-history":[{"count":9,"href":"https:\/\/www.luyouli.com\/index.php?rest_route=\/wp\/v2\/posts\/667\/revisions"}],"predecessor-version":[{"id":682,"href":"https:\/\/www.luyouli.com\/index.php?rest_route=\/wp\/v2\/posts\/667\/revisions\/682"}],"wp:attachment":[{"href":"https:\/\/www.luyouli.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=667"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.luyouli.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=667"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.luyouli.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=667"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}