taosdata/TDengine

one of vnode sync failed

Open

#23.716 aperta il 16 nov 2023

Vedi su GitHub
 (1 commento) (0 reazioni) (0 assegnatari)C (5002 fork)batch import
help wantedquestion

Metriche repository

Star
 (24.849 star)
Metriche merge PR
 (Merge medio 1g 1h) (4 PR mergiate in 30 g)

Descrizione

I have a three-node TDengine cluster and each database have three replica . One of the nodes failed and could not recover, so I rebuild node. After rejoining the cluster, the cluster began to synchronize. During the synchronization process, it was found that a vnode node failed to synchronize, reporting the following error.

11/16 09:55:02.080128 00243277 SYN WARN vgId:13, out of buffer range. index:500619097, term:60. log buffer: [-1 -1 -1, 0)
11/16 09:55:05.280023 00243277 SYN WARN vgId:13, out of buffer range. index:500623534, term:60. log buffer: [-1 -1 -1, 0)
11/16 09:55:05.455793 00243253 TSD vgId:12, tsdb end commit
11/16 09:55:05.456222 00243253 VND vgId:12, vnode info is committed
11/16 09:55:05.476152 00243253 TSD vgId:12, tsdb finish commit
11/16 09:55:05.495344 00243253 VND vgId:12, commit end
11/16 09:55:05.495362 00243253 VND vgId:10, start to commit, commitId:148 version:498216199 term: 185
11/16 09:55:05.569807 00243277 TDB ERROR unable to insert dup key. pKey: 0x7f243025863d, kLen: 22, btc: 0x7f23ebffe840, pTxn: 0x7f226c54be70
11/16 09:55:05.569828 00243277 MTA ERROR vgId:13, vnode snapshot meta write failed since Unknown error 65535
11/16 09:55:05.569832 00243277 VND ERROR vgId:13, vnode snapshot write failed since Unknown error 65535, index:100259 type:1 nData:119
11/16 09:55:05.569849 00243277 SYN ERROR vgId:13, snapshot receiver continue write failed since success, sync:follower, snap-receiver:{0x7f23cc1aa4b0 started:1 acked:100258 term:60 start-time:1700099668850 from-dnode:1, start:0 end:500598390 last-index:500598390 last-term:60 last-cfg:-1}, term:60, commit-index:-1, firstver:-1, lastver:-1, min-match:-1, snap:{last-index:-1, last-term:0}, standby:0, batch-sz:1, replicas:3, last-cfg:-1, chging:0, restore:0, quorum:2, lc-timers:{elect:4315159, hb:0}, peer:{0:-1 0, 1:-1 0, 2:-1 0}, cfg:{num:3, as:1, [hbase-rs-06.bigdata.shiqiao.com:6030, hbase-rs-07.bigdata.shiqiao.com:7030, hbase-rs-08.bigdata.shiqiao.com:6030]}
11/16 09:55:05.569864 00243277 VND ERROR vgId:13, failed to process sync msg:0x7f2430516298 type:sync-snapshot-send since success, gtid:0x0:0x11183d5d6ca3c702
11/16 09:55:05.572211 00243277 SYN WARN vgId:13, out of buffer range. index:500623719, term:60. log buffer: [-1 -1 -1, 0)
11/16 09:55:05.603510 00243254 TSD vgId:9, tsdb end commit
11/16 09:55:05.603805 00243254 VND vgId:9, vnode info is committed
11/16 09:55:05.614588 00243254 TSD vgId:9, tsdb finish commit
11/16 09:55:05.629785 00243254 VND vgId:9, commit end
11/16 09:55:07.105622 00243251 TSD vgId:8, tsdb end commit
11/16 09:55:07.105944 00243251 VND vgId:8, vnode info is committed
11/16 09:55:07.120264 00243251 TSD vgId:8, tsdb finish commit
11/16 09:55:07.140111 00243251 VND vgId:8, commit end
11/16 09:55:07.572959 00243252 TSD vgId:11, tsdb end commit
11/16 09:55:07.573277 00243252 VND vgId:11, vnode info is committed
11/16 09:55:07.590018 00243252 TSD vgId:11, tsdb finish commit
11/16 09:55:07.610833 00243252 VND vgId:11, commit end
11/16 09:55:08.772620 00243277 SYN WARN vgId:13, out of buffer range. index:500627594, term:60. log buffer: [-1 -1 -1, 0)
11/16 09:55:11.972660 00243277 SYN WARN vgId:13, out of buffer range. index:500631253, term:60. log buffer: [-1 -1 -1, 0)
11/16 09:55:14.881641 00243253 TSD vgId:10, tsdb end commit
11/16 09:55:14.882040 00243253 VND vgId:10, vnode info is committed
11/16 09:55:14.914654 00243253 TSD vgId:10, tsdb finish commit
11/16 09:55:14.957983 00243253 VND vgId:10, commit end
11/16 09:55:15.180012 00243277 SYN WARN vgId:13, out of buffer range. index:500634689, term:60. log buffer: [-1 -1 -1, 0)

the h2 node is the rebuild node

environment: OS:centos 7.9 memory:128G,CPU 40C, disk 4T,tmp 120G tdengine: version: 3.0.2.5

Guida contributor