[DocDB] Master /api/v1/tablet-servers endpoint can print wrong uptime on master leader change

Jira Link: DB-5277

Description

YB 2.17.0.0b24 OS: Linux Alma 8.7

When the masters are requested to change leader, such as when:

yb-admin -init_master_addrs=localhost:7100 master_leader_stepdown <UUID>

Is executed, the endpoint /api/v1/tablet-servers on the new master takes some time to get populated. This is likely because the tablet server heartbeat packets need to arrive at the new master that contains the information from the tablet servers, which includes their uptime.

In most of the cases, the tablet-server entries do contain the correct information, probably once it received the heartbeat packet, and because these are sent independently from the tablet servers, not all tablet server entries might show up. So far so good.

However, sometimes, for which I currently cannot find a reason, a tablet server entry can show a wrong number for uptime:

• A first appearance might show '0' (zero) for some time.
• A first appearance might show a too low uptime for some time. It should be noted that after an undefined amount of time, the entry is corrected. For this it's likely this happens when a new heartbeat packet arrives, and the entry is refreshed.

How to reproduce:

Obtain the uptime in a loop, such as:

while true; do
yb_stats --print-tablet-servers | grep Uptime
sleep 1
echo
done

And run master_leader_stepdown to move the master leader to a new master process.

This does NOT happen consistently!

Example 1 wrong uptime:

                     HB time: 0.9s, Uptime: 10737, Ram 44.78 MB
                     HB time: 0.9s, Uptime: 10732, Ram 43.28 MB
                     HB time: 0.8s, Uptime: 10734, Ram 41.45 MB

                     HB time: 0.3s, Uptime: 10737, Ram 44.78 MB
                     HB time: 0.3s, Uptime: 10737, Ram 43.28 MB
                     HB time: 0.3s, Uptime: 10739, Ram 41.45 MB
                                                                                                   << master leader switch is initiated
                                                                                                   << for a few loops the master leader is not ready

                     HB time: 0.6s, Uptime: 10742, Ram 44.78 MB << uptime correct, see before switch
                     HB time: 0.6s, Uptime: 10626, Ram 43.23 MB << uptime mysteriously lower
                     HB time: 0.6s, Uptime: 10627, Ram 41.45 MB  << uptime mysteriously lower

                     HB time: 0.6s, Uptime: 10742, Ram 44.78 MB
                     HB time: 0.6s, Uptime: 10743, Ram 43.28 MB << this uptime is corrected
                     HB time: 0.6s, Uptime: 10627, Ram 41.45 MB

                     HB time: 0.7s, Uptime: 10742, Ram 44.78 MB
                     HB time: 0.7s, Uptime: 10743, Ram 43.28 MB
                     HB time: 0.7s, Uptime: 10744, Ram 41.45 MB << now this uptime is corrected too

Example 2 uptime zero:

                     HB time: 0.8s, Uptime: 10711, Ram 44.78 MB
                     HB time: 0.8s, Uptime: 10707, Ram 43.23 MB
                     HB time: 0.8s, Uptime: 10708, Ram 41.45 MB
                                                                                                   << master leader switch is initiated
                                                                                                   << for a few loops the master leader is not ready

                     HB time: 67.6s, Uptime: 0, Ram 44.74 MB << uptime zero, high heartbeat time(?)
                     HB time: 67.6s, Uptime: 0, Ram 43.23 MB << maybe this comes from the old master, not sure
                     HB time: 67.6s, Uptime: 0, Ram 41.45 MB

                     HB time: 0.4s, Uptime: 0, Ram 0 B << all uptime still zero
                     HB time: 0.5s, Uptime: 0, Ram 0 B
                     HB time: 0.4s, Uptime: 0, Ram 0 B

                     HB time: 0.5s, Uptime: 10717, Ram 44.78 MB << one heartbeat packet received
                     HB time: 0.5s, Uptime: 0, Ram 0 B
                     HB time: 0.4s, Uptime: 0, Ram 0 B

                     HB time: 0.5s, Uptime: 10717, Ram 44.78 MB
                     HB time: 0.5s, Uptime: 10717, Ram 43.28 MB << two received, etc.
                     HB time: 0.5s, Uptime: 0, Ram 0 B

                     HB time: 0.5s, Uptime: 10717, Ram 44.78 MB
                     HB time: 0.6s, Uptime: 10717, Ram 43.28 MB
                     HB time: 0.5s, Uptime: 10719, Ram 41.45 MB

For understanding changes in the cluster and using the available endpoints to diagnose that, this intermediate state of wrong information makes diagnosis really hard.