help wantedquestion
仓库指标
- Star
- (24,849 star)
- PR 合并指标
- (平均合并 1天 1小时) (30 天内合并 4 个 PR)
描述
I have a three-node TDengine cluster and each database have three replica . One of the nodes failed and could not recover, so I rebuild node. After rejoining the cluster, the cluster began to synchronize. During the synchronization process, it was found that a vnode node failed to synchronize, reporting the following error.
11/16 09:55:02.080128 00243277 SYN WARN vgId:13, out of buffer range. index:500619097, term:60. log buffer: [-1 -1 -1, 0)
11/16 09:55:05.280023 00243277 SYN WARN vgId:13, out of buffer range. index:500623534, term:60. log buffer: [-1 -1 -1, 0)
11/16 09:55:05.455793 00243253 TSD vgId:12, tsdb end commit
11/16 09:55:05.456222 00243253 VND vgId:12, vnode info is committed
11/16 09:55:05.476152 00243253 TSD vgId:12, tsdb finish commit
11/16 09:55:05.495344 00243253 VND vgId:12, commit end
11/16 09:55:05.495362 00243253 VND vgId:10, start to commit, commitId:148 version:498216199 term: 185
11/16 09:55:05.569807 00243277 TDB ERROR unable to insert dup key. pKey: 0x7f243025863d, kLen: 22, btc: 0x7f23ebffe840, pTxn: 0x7f226c54be70
11/16 09:55:05.569828 00243277 MTA ERROR vgId:13, vnode snapshot meta write failed since Unknown error 65535
11/16 09:55:05.569832 00243277 VND ERROR vgId:13, vnode snapshot write failed since Unknown error 65535, index:100259 type:1 nData:119
11/16 09:55:05.569849 00243277 SYN ERROR vgId:13, snapshot receiver continue write failed since success, sync:follower, snap-receiver:{0x7f23cc1aa4b0 started:1 acked:100258 term:60 start-time:1700099668850 from-dnode:1, start:0 end:500598390 last-index:500598390 last-term:60 last-cfg:-1}, term:60, commit-index:-1, firstver:-1, lastver:-1, min-match:-1, snap:{last-index:-1, last-term:0}, standby:0, batch-sz:1, replicas:3, last-cfg:-1, chging:0, restore:0, quorum:2, lc-timers:{elect:4315159, hb:0}, peer:{0:-1 0, 1:-1 0, 2:-1 0}, cfg:{num:3, as:1, [hbase-rs-06.bigdata.shiqiao.com:6030, hbase-rs-07.bigdata.shiqiao.com:7030, hbase-rs-08.bigdata.shiqiao.com:6030]}
11/16 09:55:05.569864 00243277 VND ERROR vgId:13, failed to process sync msg:0x7f2430516298 type:sync-snapshot-send since success, gtid:0x0:0x11183d5d6ca3c702
11/16 09:55:05.572211 00243277 SYN WARN vgId:13, out of buffer range. index:500623719, term:60. log buffer: [-1 -1 -1, 0)
11/16 09:55:05.603510 00243254 TSD vgId:9, tsdb end commit
11/16 09:55:05.603805 00243254 VND vgId:9, vnode info is committed
11/16 09:55:05.614588 00243254 TSD vgId:9, tsdb finish commit
11/16 09:55:05.629785 00243254 VND vgId:9, commit end
11/16 09:55:07.105622 00243251 TSD vgId:8, tsdb end commit
11/16 09:55:07.105944 00243251 VND vgId:8, vnode info is committed
11/16 09:55:07.120264 00243251 TSD vgId:8, tsdb finish commit
11/16 09:55:07.140111 00243251 VND vgId:8, commit end
11/16 09:55:07.572959 00243252 TSD vgId:11, tsdb end commit
11/16 09:55:07.573277 00243252 VND vgId:11, vnode info is committed
11/16 09:55:07.590018 00243252 TSD vgId:11, tsdb finish commit
11/16 09:55:07.610833 00243252 VND vgId:11, commit end
11/16 09:55:08.772620 00243277 SYN WARN vgId:13, out of buffer range. index:500627594, term:60. log buffer: [-1 -1 -1, 0)
11/16 09:55:11.972660 00243277 SYN WARN vgId:13, out of buffer range. index:500631253, term:60. log buffer: [-1 -1 -1, 0)
11/16 09:55:14.881641 00243253 TSD vgId:10, tsdb end commit
11/16 09:55:14.882040 00243253 VND vgId:10, vnode info is committed
11/16 09:55:14.914654 00243253 TSD vgId:10, tsdb finish commit
11/16 09:55:14.957983 00243253 VND vgId:10, commit end
11/16 09:55:15.180012 00243277 SYN WARN vgId:13, out of buffer range. index:500634689, term:60. log buffer: [-1 -1 -1, 0)
the h2 node is the rebuild node
environment: OS:centos 7.9 memory:128G,CPU 40C, disk 4T,tmp 120G tdengine: version: 3.0.2.5