3-node tdengine 3.2.2.0 cluster in docker swarm cannot elect new leader after failure and client cannot connect to followers

Bug 描述 在3台虚拟机上使用docker swarm部署了一个3节点集群，数据插入同步正常，当有一个节点挂掉后，就会出现Sync leader is unreachable错误。当挂掉的节点恢复正常后，集群功能也会恢复。 截图 日志

07/26 07:11:12.671898 00000153 SYN vgId:1, begin election, sync:follower, term:1, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:0, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:144, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:12.671961 00000153 SYN vgId:1, become candidate from follower. term:1, commit index:24, last index:24
07/26 07:11:12.673020 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:2
07/26 07:11:12.673620 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:2
07/26 07:11:12.674124 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:2
07/26 07:11:13.584294 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-1:6030, td-2:6030, td-3:6030}, inUse:0
07/26 07:11:18.046860 00000153 SYN vgId:1, begin election, sync:candidate, term:2, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:1, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:145, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:18.048060 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:3
07/26 07:11:18.048654 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:3
07/26 07:11:18.049785 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:3
07/26 07:11:18.784752 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-2:6030, td-3:6030, td-1:6030}, inUse:0
07/26 07:11:21.753837 00000090 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb40033a0 failed to connect to td-1:6030, reason: operation canceled, gtid:0x0:0x7bc40ede11100190
07/26 07:11:21.871924 00000153 SYN vgId:1, begin election, sync:candidate, term:3, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:2, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:146, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:21.872887 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:4
07/26 07:11:21.873610 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:4
07/26 07:11:21.874227 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:4
07/26 07:11:22.106021 00000153 SYN vgId:1, recv sync-request-vote from dnode:3, {term:2, last-index:24, last-term:1}, granted:0, sync:candidate, term:4, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:3, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:147, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:22.106062 00000153 SYN vgId:1, send sync-request-vote-reply to dnode:3 {term:4, grant:0}, , sync:candidate, term:4, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:3, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:147, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:23.985046 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-3:6030, td-1:6030, td-2:6030}, inUse:0
07/26 07:11:26.506896 00000153 SYN vgId:1, begin election, sync:candidate, term:4, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:3, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:147, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:26.507993 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:5
07/26 07:11:26.508917 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:5
07/26 07:11:26.510059 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:5
07/26 07:11:26.958776 00000089 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb8003570 failed to connect to td-1:6030, reason: operation canceled, gtid:0x0:0x7bc40ede12500193
07/26 07:11:27.299890 00000153 SYN vgId:1, recv sync-request-vote from dnode:3, {term:3, last-index:24, last-term:1}, granted:0, sync:candidate, term:5, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:4, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:148, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:27.299911 00000153 SYN vgId:1, send sync-request-vote-reply to dnode:3 {term:5, grant:0}, , sync:candidate, term:5, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:4, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:148, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:29.185299 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-1:6030, td-2:6030, td-3:6030}, inUse:0
07/26 07:11:29.510458 00000106 RPC ERROR http-report failed to resolving domain names: buildkitsandbox
07/26 07:11:29.510472 00000106 RPC ERROR http-report failed to report, reason: success, addr: buildkitsandbox:6043
07/26 07:11:31.051894 00000153 SYN vgId:1, begin election, sync:candidate, term:5, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:4, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:148, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:31.053231 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:6
07/26 07:11:31.053809 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:6
07/26 07:11:31.054343 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:6
07/26 07:11:34.385570 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-2:6030, td-3:6030, td-1:6030}, inUse:0
07/26 07:11:34.761902 00000153 SYN vgId:1, begin election, sync:candidate, term:6, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:5, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:149, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:34.763180 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:7
07/26 07:11:34.764151 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:7
07/26 07:11:34.764883 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:7
07/26 07:11:39.585950 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-3:6030, td-1:6030, td-2:6030}, inUse:0
07/26 07:11:40.526942 00000153 SYN vgId:1, begin election, sync:candidate, term:7, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:6, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:150, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:40.528318 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:8
07/26 07:11:40.528831 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:8
07/26 07:11:40.529297 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:8
07/26 07:11:41.786954 00000090 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb4005a90 failed to connect to td-1:6030, reason: operation canceled
07/26 07:11:41.787397 00000090 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb4007d80 failed to connect to td-3:6030, reason: operation canceled
07/26 07:11:42.151450 00000153 SYN vgId:1, recv sync-request-vote from dnode:3, {term:5, last-index:24, last-term:1}, granted:0, sync:candidate, term:8, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:7, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:151, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:42.151491 00000153 SYN vgId:1, send sync-request-vote-reply to dnode:3 {term:8, grant:0}, , sync:candidate, term:8, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:7, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:151, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:43.721846 00000153 SYN vgId:1, begin election, sync:candidate, term:8, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:7, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:151, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:43.722654 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:9
07/26 07:11:43.723146 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:9
07/26 07:11:43.723670 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:9
07/26 07:11:44.786352 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-1:6030, td-2:6030, td-3:6030}, inUse:0
07/26 07:11:46.993861 00000089 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb80059d0 failed to connect to td-1:6030, reason: operation canceled
07/26 07:11:47.961862 00000153 SYN vgId:1, begin election, sync:candidate, term:9, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:8, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:152, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:47.963134 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:10
07/26 07:11:47.964522 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:10
07/26 07:11:47.965145 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:10
07/26 07:11:49.986946 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-2:6030, td-3:6030, td-1:6030}, inUse:0
07/26 07:11:52.756877 00000153 SYN vgId:1, begin election, sync:candidate, term:10, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:9, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:153, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:52.758027 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:11
07/26 07:11:52.758951 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:11
07/26 07:11:52.760100 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:11
07/26 07:11:55.187502 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-3:6030, td-1:6030, td-2:6030}, inUse:0
07/26 07:11:57.008758 00000089 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb8007d70 failed to connect to td-1:6030, reason: operation canceled
07/26 07:11:57.009307 00000089 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb8009060 failed to connect to td-3:6030, reason: operation canceled
07/26 07:11:58.466947 00000153 SYN vgId:1, begin election, sync:candidate, term:11, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:10, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:154, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:11:58.467974 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:12
07/26 07:11:58.469419 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:12
07/26 07:11:58.470097 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:12
07/26 07:11:59.536906 00000106 RPC ERROR http-report failed to resolving domain names: buildkitsandbox
07/26 07:11:59.536921 00000106 RPC ERROR http-report failed to report, reason: success, addr: buildkitsandbox:6043
07/26 07:12:00.388000 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-1:6030, td-2:6030, td-3:6030}, inUse:0
07/26 07:12:01.821741 00000090 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb40033a0 failed to connect to td-1:6030, reason: operation canceled, gtid:0x0:0x7bc40ede18b001a4
07/26 07:12:01.822196 00000090 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb4008f00 failed to connect to td-3:6030, reason: operation canceled
07/26 07:12:01.822665 00000090 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb400a260 failed to connect to td-3:6030, reason: operation canceled
07/26 07:12:01.823200 00000090 RPC WARN update ip of fqdn:td-3, old: 10.0.1.244, new: 10.0.1.243
07/26 07:12:01.823207 00000090 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb400b5c0 failed to connect to td-3:6030, reason: operation canceled
07/26 07:12:02.251828 00000153 SYN vgId:1, begin election, sync:candidate, term:12, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:11, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:155, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:02.252733 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:13
07/26 07:12:02.253233 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:13
07/26 07:12:02.253676 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:13
07/26 07:12:05.588485 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-2:6030, td-3:6030, td-1:6030}, inUse:0
07/26 07:12:05.801813 00000153 SYN vgId:1, begin election, sync:candidate, term:13, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:12, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:156, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:05.802964 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:14
07/26 07:12:05.803935 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:14
07/26 07:12:05.805252 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:14
07/26 07:12:09.026854 00000153 SYN vgId:1, begin election, sync:candidate, term:14, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:13, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:157, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:09.027664 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:15
07/26 07:12:09.028280 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:15
07/26 07:12:09.028990 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:15
07/26 07:12:10.788752 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-3:6030, td-1:6030, td-2:6030}, inUse:0
07/26 07:12:11.838877 00000090 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb400c900 failed to connect to td-1:6030, reason: operation canceled, gtid:0x0:0x7bc40ede14e0019a
07/26 07:12:13.351994 00000153 SYN vgId:1, begin election, sync:candidate, term:15, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:14, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:158, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:13.353178 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:16
07/26 07:12:13.354088 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:16
07/26 07:12:13.355016 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:16
07/26 07:12:15.989134 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-1:6030, td-2:6030, td-3:6030}, inUse:0
07/26 07:12:16.366847 00000153 SYN vgId:1, begin election, sync:candidate, term:16, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:15, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:159, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:16.368332 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:17
07/26 07:12:16.368910 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:17
07/26 07:12:16.369433 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:17
07/26 07:12:20.756765 00000153 SYN vgId:1, begin election, sync:candidate, term:17, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:16, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:160, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:20.757619 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:18
07/26 07:12:20.758263 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:18
07/26 07:12:20.758911 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:18
07/26 07:12:21.189611 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-2:6030, td-3:6030, td-1:6030}, inUse:0
07/26 07:12:21.856867 00000090 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb400dc80 failed to connect to td-1:6030, reason: operation canceled
07/26 07:12:25.001869 00000153 SYN vgId:1, begin election, sync:candidate, term:18, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:17, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:161, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:25.002887 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:19
07/26 07:12:25.003829 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:19
07/26 07:12:25.004395 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:19
07/26 07:12:26.390065 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-3:6030, td-1:6030, td-2:6030}, inUse:0
07/26 07:12:27.051857 00000089 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb8003570 failed to connect to td-1:6030, reason: operation canceled
07/26 07:12:27.053005 00000089 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb800a380 failed to connect to td-3:6030, reason: operation canceled, gtid:0x0:0x7bc40ede1b3001ad
07/26 07:12:29.557331 00000106 RPC ERROR http-report failed to resolving domain names: buildkitsandbox
07/26 07:12:29.557345 00000106 RPC ERROR http-report failed to report, reason: success, addr: buildkitsandbox:6043
07/26 07:12:30.986973 00000153 SYN vgId:1, begin election, sync:candidate, term:19, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:18, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:162, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:30.988104 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:20
07/26 07:12:30.989073 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:20
07/26 07:12:30.989919 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:20
07/26 07:12:31.590418 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-1:6030, td-2:6030, td-3:6030}, inUse:0
07/26 07:12:31.873890 00000090 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb4005a90 failed to connect to td-1:6030, reason: host is unreachable, gtid:0x0:0x7bc40ede13900196
07/26 07:12:34.121923 00000153 SYN vgId:1, begin election, sync:candidate, term:20, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:19, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:163, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:34.123239 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:21
07/26 07:12:34.124104 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:21
07/26 07:12:34.125597 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:21
07/26 07:12:36.790743 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-2:6030, td-3:6030, td-1:6030}, inUse:0
07/26 07:12:37.068875 00000089 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb800b630 failed to connect to td-1:6030, reason: operation canceled
07/26 07:12:37.069601 00000089 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb800c930 failed to connect to td-3:6030, reason: operation canceled
07/26 07:12:38.316793 00000153 SYN vgId:1, begin election, sync:candidate, term:21, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:20, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:164, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:38.317948 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:22
07/26 07:12:38.318450 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:22
07/26 07:12:38.318843 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:22
07/26 07:12:41.991049 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-3:6030, td-1:6030, td-2:6030}, inUse:0
07/26 07:12:43.256772 00000153 SYN vgId:1, begin election, sync:candidate, term:22, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:21, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:165, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:43.257620 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:23
07/26 07:12:43.258172 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:23
07/26 07:12:43.258766 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:23
07/26 07:12:47.091285 00000089 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb800dcf0 failed to connect to td-1:6030, reason: operation canceled
07/26 07:12:47.091663 00000089 RPC WARN update ip of fqdn:td-3, old: 10.0.1.244, new: 10.0.1.243
07/26 07:12:47.091669 00000089 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb800f030 failed to connect to td-3:6030, reason: operation canceled
07/26 07:12:47.092107 00000089 RPC WARN update ip of fqdn:td-3, old: 10.0.1.243, new: 10.0.1.244
07/26 07:12:47.092114 00000089 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb8010370 failed to connect to td-3:6030, reason: operation canceled, gtid:0x0:0x7bc40ede176001a1
07/26 07:12:47.092537 00000089 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb80116f0 failed to connect to td-3:6030, reason: operation canceled
07/26 07:12:47.191369 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-1:6030, td-2:6030, td-3:6030}, inUse:0
07/26 07:12:47.296831 00000153 SYN vgId:1, begin election, sync:candidate, term:23, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:22, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:166, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:47.298279 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:24
07/26 07:12:47.299017 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:24
07/26 07:12:47.299610 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:24
07/26 07:12:51.311771 00000153 SYN vgId:1, begin election, sync:candidate, term:24, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:23, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:167, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:51.312725 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:25
07/26 07:12:51.313332 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:25
07/26 07:12:51.313773 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:25
07/26 07:12:52.391689 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-2:6030, td-3:6030, td-1:6030}, inUse:0
07/26 07:12:55.736803 00000153 SYN vgId:1, begin election, sync:candidate, term:25, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:24, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:168, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:12:55.738175 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:26
07/26 07:12:55.738565 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:26
07/26 07:12:55.738894 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:26
07/26 07:12:57.113315 00000089 RPC WARN update ip of fqdn:td-3, old: 10.0.1.244, new: 10.0.1.243
07/26 07:12:57.591959 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-3:6030, td-1:6030, td-2:6030}, inUse:0
07/26 07:12:59.584129 00000106 RPC ERROR http-report failed to resolving domain names: buildkitsandbox
07/26 07:12:59.584144 00000106 RPC ERROR http-report failed to report, reason: success, addr: buildkitsandbox:6043
07/26 07:13:00.526985 00000153 SYN vgId:1, begin election, sync:candidate, term:26, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:25, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:169, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:13:00.528167 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:27
07/26 07:13:00.528875 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:27
07/26 07:13:00.529653 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:27
07/26 07:13:02.792459 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-1:6030, td-2:6030, td-3:6030}, inUse:0
07/26 07:13:04.346775 00000153 SYN vgId:1, begin election, sync:candidate, term:27, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:26, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:170, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:13:04.348357 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:28
07/26 07:13:04.348998 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:28
07/26 07:13:04.349648 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:28
07/26 07:13:07.132353 00000089 RPC WARN update ip of fqdn:td-3, old: 10.0.1.243, new: 10.0.1.244
07/26 07:13:07.992773 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-2:6030, td-3:6030, td-1:6030}, inUse:0
07/26 07:13:09.296892 00000153 SYN vgId:1, begin election, sync:candidate, term:28, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:27, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:171, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:13:09.297935 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:29
07/26 07:13:09.298867 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:29
07/26 07:13:09.299523 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:29
07/26 07:13:11.936847 00000090 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb4010590 failed to connect to td-1:6030, reason: operation canceled
07/26 07:13:11.937626 00000090 RPC WARN update ip of fqdn:td-3, old: 10.0.1.243, new: 10.0.1.244
07/26 07:13:11.937635 00000090 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb40118e0 failed to connect to td-3:6030, reason: operation canceled, gtid:0x0:0x7bc40ede1f0001b6
07/26 07:13:11.937839 00000090 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb4012b90 failed to connect to td-2:6030, reason: operation canceled, gtid:0x0:0x7bc40ede219001be
07/26 07:13:13.193082 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-3:6030, td-1:6030, td-2:6030}, inUse:0
07/26 07:13:13.242052 00000153 SYN vgId:1, begin election, sync:candidate, term:29, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:28, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:172, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:13:13.244308 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:30
07/26 07:13:13.245352 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:30
07/26 07:13:13.246224 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:30
07/26 07:13:17.347177 00000153 SYN vgId:1, begin election, sync:candidate, term:30, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:29, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:173, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:13:17.348495 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:31
07/26 07:13:17.349318 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:31
07/26 07:13:17.350127 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:31
07/26 07:13:18.393719 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-1:6030, td-2:6030, td-3:6030}, inUse:0
07/26 07:13:21.081951 00000153 SYN vgId:1, begin election, sync:candidate, term:31, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:30, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:174, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:13:21.083034 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:32
07/26 07:13:21.084474 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:32
07/26 07:13:21.085129 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:32
07/26 07:13:21.956898 00000090 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb4013e90 failed to connect to td-1:6030, reason: operation canceled
07/26 07:13:23.594158 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-2:6030, td-3:6030, td-1:6030}, inUse:0
07/26 07:13:26.561810 00000153 SYN vgId:1, begin election, sync:candidate, term:32, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:31, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:175, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:13:26.562702 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:33
07/26 07:13:26.563229 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:33
07/26 07:13:26.563720 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:33
07/26 07:13:27.161962 00000089 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb80059d0 failed to connect to td-1:6030, reason: operation canceled, gtid:0x0:0x7bc40ede1b3001ad
07/26 07:13:28.799999 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-3:6030, td-1:6030, td-2:6030}, inUse:0
07/26 07:13:29.621911 00000106 RPC ERROR http-report failed to resolving domain names: buildkitsandbox
07/26 07:13:29.623084 00000106 RPC ERROR http-report failed to report, reason: success, addr: buildkitsandbox:6043
07/26 07:13:30.381933 00000153 SYN vgId:1, begin election, sync:candidate, term:33, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:32, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:176, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:13:30.382995 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:34
07/26 07:13:30.384398 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:34
07/26 07:13:30.385280 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:34
07/26 07:13:31.976926 00000090 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb4015210 failed to connect to td-1:6030, reason: operation canceled, gtid:0x0:0x7bc40ede1c7001b0
07/26 07:13:34.004177 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-1:6030, td-2:6030, td-3:6030}, inUse:0
07/26 07:13:35.286874 00000153 SYN vgId:1, begin election, sync:candidate, term:34, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:33, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:177, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:13:35.288756 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:35
07/26 07:13:35.289364 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:35
07/26 07:13:35.289922 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:35
07/26 07:13:37.176902 00000089 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb8007ca0 failed to connect to td-1:6030, reason: operation canceled, gtid:0x0:0x7bc40ede176001a1
07/26 07:13:39.204510 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-2:6030, td-3:6030, td-1:6030}, inUse:0
07/26 07:13:39.746929 00000153 SYN vgId:1, begin election, sync:candidate, term:35, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:34, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:178, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:13:39.748143 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:36
07/26 07:13:39.748649 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:36
07/26 07:13:39.749157 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:36
07/26 07:13:41.992001 00000090 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb4016590 failed to connect to td-1:6030, reason: operation canceled
07/26 07:13:41.993941 00000090 RPC WARN update ip of fqdn:td-3, old: 10.0.1.244, new: 10.0.1.243
07/26 07:13:41.993953 00000090 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb40178d0 failed to connect to td-3:6030, reason: operation canceled
07/26 07:13:41.994128 00000090 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb4018c10 failed to connect to td-2:6030, reason: operation canceled, gtid:0x0:0x7bc40ede19f001aa
07/26 07:13:41.995034 00000090 RPC WARN update ip of fqdn:td-3, old: 10.0.1.243, new: 10.0.1.244
07/26 07:13:41.995047 00000090 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb4019f90 failed to connect to td-3:6030, reason: operation canceled
07/26 07:13:41.996714 00000090 RPC WARN update ip of fqdn:td-3, old: 10.0.1.244, new: 10.0.1.243
07/26 07:13:44.404916 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-3:6030, td-1:6030, td-2:6030}, inUse:0
07/26 07:13:45.111916 00000153 SYN vgId:1, begin election, sync:candidate, term:36, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:35, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:179, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:13:45.113829 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:37
07/26 07:13:45.114640 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:37
07/26 07:13:45.115523 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:37
07/26 07:13:47.196970 00000089 RPC ERROR DND-C msg status failed to send, conn 0x7f7fb80080c0 failed to connect to td-1:6030, reason: operation canceled, gtid:0x0:0x7bc40ede204001bb
07/26 07:13:47.197446 00000089 RPC WARN update ip of fqdn:td-3, old: 10.0.1.244, new: 10.0.1.243
07/26 07:13:47.197451 00000089 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb8008310 failed to connect to td-3:6030, reason: operation canceled
07/26 07:13:49.605428 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-1:6030, td-2:6030, td-3:6030}, inUse:0
07/26 07:13:51.072011 00000153 SYN vgId:1, begin election, sync:candidate, term:37, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:36, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:180, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:13:51.072910 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:38
07/26 07:13:51.073737 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:38
07/26 07:13:51.074345 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:38
07/26 07:13:54.805861 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-2:6030, td-3:6030, td-1:6030}, inUse:0
07/26 07:13:55.641899 00000153 SYN vgId:1, begin election, sync:candidate, term:38, commit-index:24, applied-index:24, first-ver:0, last-ver:24, min:0, snap:13, snap-term:1, elect-times:37, as-leader-times:0, cfg-ch-times:2, hb-slow:0, hbr-slow:0, aq-items:-1, snaping:-1, replicas:3, last-cfg:20, chging:0, restore:1, quorum:2, elect-lc-timer:181, hb:0, buffer:[13 24 24, 25), repl-mgrs:{0:0 [0 0, 0), 1:0 [0 0, 0), 2:0 [0 0, 0)}, members:{num:3, as:1, [td-1:6030, td-2:6030, td-3:6030]}, hb:{0:1721977868138,1:1721977806120,2:1721977806120}, hb-reply:{0:1721977806120,1:1721977806120,2:1721977806120}
07/26 07:13:55.643974 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:39
07/26 07:13:55.644860 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:39
07/26 07:13:55.645469 00000153 SYN vgId:1, succeed to write raft store file:/var/lib/taos//mnode/sync/raft_store.json, term:39
07/26 07:13:57.211878 00000089 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb80085b0 failed to connect to td-1:6030, reason: operation canceled
07/26 07:13:57.212462 00000089 RPC WARN update ip of fqdn:td-3, old: 10.0.1.243, new: 10.0.1.244
07/26 07:13:57.212469 00000089 RPC ERROR DND-C batch msg failed to send, conn 0x7f7fb8008850 failed to connect to td-3:6030, reason: operation canceled
07/26 07:13:59.668868 00000106 RPC ERROR http-report failed to resolving domain names: buildkitsandbox
07/26 07:13:59.668887 00000106 RPC ERROR http-report failed to report, reason: success, addr: buildkitsandbox:6043
07/26 07:14:00.006300 00000099 DND ERROR failed to send status req since Operation timeout, epSet:{td-3:6030, td-1:6030, td-2:6030}, inUse:0