Multiple peers sections lose stick tables on reload (shared dont_stop state)

The shared latch is the function-static dont_stop in __process_stopping_peer_sync() in src/peers.c (~line 3797); struct peers lives in include/haproxy/peers-t.h and is calloc'd in src/cfgparse-peers.c. Build a 3.4.6 tree and run the issue's reproduce.py, which reloads the master three times and prints show table first/marker and second/marker. The proposed change moves the flag into struct peers so each section keeps its own jobs hold, table refcounts and local handover session. Done when both markers survive every reload.

Detailed Description of the Problem

With two independent peers sections, a graceful master-worker reload loses the marker in one section. The other section transfers its marker successfully. This reproduces with unmodified HAProxy 3.4.6, one thread, two 100-entry tables containing one entry each, no remote peers, no TLS, no frontend traffic, and no configuration changes between reloads.

Three consecutive reloads of the minimal configuration below produced:

first/marker:  used:0
second/marker: used:1, gpc0=1

The new worker reports last_status=NONE last_hdshk=<NEVER> for the first section's local peer, and last_status=ESTA for the second. Both sections finish initial synchronization before each reload. Marker presence is verified immediately before signaling and checked again one second afterward.

This was first observed on 3.4.3 with two sections, one using mTLS and remote peers and the other local-only. Reloading three nodes at the same time can empty the replicated tables on all three. Reducing the configuration to one section makes the local handover work, including with mTLS. The single-thread reproduction below shows that no thread race is needed.

Expected Behavior

Each peers section should independently hand its stick-table contents to the new worker on a graceful reload. Completion or failure in one section should not release another section's shutdown hold.

Steps to Reproduce the Behavior

1. Save the configuration below as haproxy.cfg and the following script as reproduce.py in an empty directory. Requires Python 3 and available loopback ports 19883/19884 and /tmp/haproxy-peer-reload.sock.
2. Run python3 reproduce.py /absolute/path/to/haproxy haproxy.log with an unmodified 3.4.6 build.
3. Observe that first/marker is empty after each reload while second/marker retains its marker. The script waits for both sections' resync-finished flags before each reload and terminates its test process group afterward.

"""Reproduce local handover loss without remote peers or TLS."""
import os
import re
import signal
import socket
import subprocess
import sys
import time
from pathlib import Path


def cli(command):
    """Return a runtime command response from the test socket."""
    with socket.socket(socket.AF_UNIX) as client:
        client.settimeout(2)
        client.connect('/tmp/haproxy-peer-reload.sock')
        client.sendall((command + '\n').encode())
        result = b''
        while chunk := client.recv(65536):
            result += chunk
        return result.decode()


with open(sys.argv[2], 'w') as log:
    master = subprocess.Popen([sys.argv[1], '-W', '-db', '-f', 'haproxy.cfg'], stdout=log, stderr=log, start_new_session=True)
    try:
        time.sleep(1)
        for iteration in range(3):
            deadline = time.monotonic() + 30
            while True:
                flags = re.findall(r'disabled=0 flags=(0x[0-9a-f]+)', cli('show peers'))
                if len(flags) == 2 and all(int(f, 16) & 3 == 3 for f in flags):
                    break
                if time.monotonic() > deadline:
                    raise RuntimeError('Peer synchronization did not complete')
                time.sleep(.1)
            for table in ['first/marker', 'second/marker']:
                assert not cli(f'set table {table} key 192.0.2.1 data.gpc0 1').strip()
                assert 'gpc0=1' in cli(f'show table {table}')
            os.kill(master.pid, signal.SIGUSR2)
            time.sleep(1)
            print(f'Round {iteration + 1}', flush=True)
            for table in ['first/marker', 'second/marker']:
                print(cli(f'show table {table}'), flush=True)
            print(cli('show peers'), flush=True)
    finally:
        os.killpg(master.pid, signal.SIGTERM)
        master.wait(timeout=5)

Do you have any idea what may have caused this?

__process_stopping_peer_sync() in src/peers.c has a function-static int dont_stop (line 3797 in 3.4.6). It is shared by every section, although the surrounding code operates on one struct peers at a time. The variable gates acquiring and releasing the jobs hold, releasing the per-table refcounts taken at table registration, initializing the local resync timeout and reconnect, and creating the local handover session. Whichever section's sync task runs first owns the latch: the second section skips its own setup, and either section can release the other's hold. In multithreaded configurations the two sync tasks can also run on different threads.

Do you have an idea how to solve the issue?

Moving the latch into struct peers fixes the minimal reproducer. The structure is allocated with calloc() in cfgparse-peers.c, so the new field starts at zero, and only the section's own sync task reads or writes it, so this introduces no new shared state. This is a proposed fix for review, not an upstream-approved patch.

--- a/include/haproxy/peers-t.h
+++ b/include/haproxy/peers-t.h
@@ -182,6 +182,7 @@
 
 
 struct peers {
+	int dont_stop; /* This section holds a job during reload handover. */
 	char *id;                       /* peer section name */
 	struct task *sync_task;         /* main sync task */
 	struct sig_handler *sighandler; /* signal handler */
--- a/src/peers.c
+++ b/src/peers.c
@@ -3794,7 +3794,6 @@
 {
 	struct peer *peer;
 	struct shared_table *st;
-	static int dont_stop = 0;
 
 	/* For each peer */
 	for (peer = peers->remote; peer; peer = peer->next) {
@@ -3809,7 +3808,7 @@
 		 */
 		peer->flags &= ~PEER_F_WAIT_SYNCTASK_ACK;
 
-		if ((state & TASK_WOKEN_SIGNAL) && !dont_stop) {
+		if ((state & TASK_WOKEN_SIGNAL) && !peers->dont_stop) {
 			/* we're killing a connection, we must apply a random delay before
 			 * retrying otherwise the other end will do the same and we can loop
 			 * for a while.
@@ -3826,10 +3825,10 @@
 
 	/* We've just received the signal */
 	if (state & TASK_WOKEN_SIGNAL) {
-		if (!dont_stop) {
+		if (!peers->dont_stop) {
 			/* add DO NOT STOP flag if not present */
 			_HA_ATOMIC_INC(&jobs);
-			dont_stop = 1;
+			peers->dont_stop = 1;
 
 			/* Set resync timeout for the local peer and request a immediate reconnect */
 			peers->resync_timeout = tick_add(now_ms, MS_TO_TICKS(PEER_RESYNC_TIMEOUT));
@@ -3840,10 +3839,10 @@
 	peer = peers->local;
 	HA_SPIN_LOCK(PEER_LOCK, &peer->lock);
 	if (peer->flags & PEER_F_LOCAL_TEACH_COMPLETE) {
-		if (dont_stop) {
+		if (peers->dont_stop) {
 			/* resync of new process was complete, current process can die now */
 			_HA_ATOMIC_DEC(&jobs);
-			dont_stop = 0;
+			peers->dont_stop = 0;
 			for (st = peer->tables; st ; st = st->next)
 				HA_ATOMIC_DEC(&st->table->refcnt);
 		}
@@ -3873,17 +3872,17 @@
 			}
 			else  {
 				/* connect to the local peer if we must push a local sync */
-				if (dont_stop) {
+				if (peers->dont_stop) {
 					peer_session_create(peers, peer);
 				}
 			}
 		}
 		else {
 			/* Other error cases */
-			if (dont_stop) {
+			if (peers->dont_stop) {
 				/* unable to resync new process, current process can die now */
 				_HA_ATOMIC_DEC(&jobs);
-				dont_stop = 0;
+				peers->dont_stop = 0;
 				for (st = peer->tables; st ; st = st->next)
 					HA_ATOMIC_DEC(&st->table->refcnt);
 			}

What is your configuration?

global
  localpeer lab
  nbthread 1
  stats socket /tmp/haproxy-peer-reload.sock level admin
defaults
  mode tcp
  timeout connect 1s
  timeout client 5s
  timeout server 5s
peers first
  peer lab 127.0.0.1:19883
  table marker type ip size 100 expire 1h store gpc0
peers second
  peer lab 127.0.0.1:19884
  table marker type ip size 100 expire 1h store gpc0

Output of haproxy -vv

Unmodified 3.4.6 built with make TARGET=linux-glibc USE_OPENSSL= USE_LUA= USE_PCRE2= for the minimal reproduction. The original failure also reproduces with a full 3.4.3 AWS-LC/QUIC/Lua/PCRE2 build.

HAProxy version 3.4.6-56332c5 2026/09/28 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2031.
Known bugs: http://www.haproxy.org/bugs/bugs-3.4.6.html
Running on: Linux 6.12.107+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.107-1 (2026-08-29) x86_64
Build options : 
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv -fvect-cost-model=very-cheap
  OPTIONS = USE_OPENSSL= USE_LUA= USE_PCRE2=
  DEBUG   = 

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO +HAVE_TCP_MD5SIG -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER -OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ -SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY +TRACE -WURFL -ZLIB
Detected feature list :

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=8).
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 14.2.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] comp-req
	[COMP] comp-res
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

Last Outputs and Backtraces

No crash is required. The old worker exits normally. Representative local-peer states after reload:

first:  app_state=STOPPED last_status=NONE last_hdshk=<NEVER>
second: app_state=STOPPED last_status=ESTA last_hdshk=0s

Additional Information

A patched 3.4.6 was also built with AWS-LC 5.11.0, QUIC, Lua, PCRE2 JIT and DEBUG_STRICT. Both markers survived five single-thread and five eight-thread local reloads. In five simultaneous three-node mTLS reload rounds, two replicated tables and a separate local-only table kept their markers on all three nodes one second after each reload (45 table checks). Each round waited for all nodes to finish initial synchronization first.

Field results: a 3.4.6 build carrying this patch together with the fixes proposed in #3518, #3519, #3520 and #3521 was deployed to a four-node production cluster on 2026-10-02. Each node has two peers sections, the nodes talk over mTLS, and the replicated tables hold 1.3 to 2.4 million entries each. After deployment, reloading the nodes one at a time and then all four at once preserved every checked entry and test marker. The local handovers took 15 to 33 seconds and the new worker sent three to six progress heartbeats (#3521) during each.

• Related #1216 concerns rapid reloads and initial synchronization; this reproduction explicitly waits for synchronization and has no remote peer.
• #842 concerned reloads triggered by service-discovery configuration changes; this configuration validates and remains unchanged between reloads.
• The same function-static latch is present in the 3.2.6 source and in the development branch as of 2026-10-02. Those are source inspections, not runtime tests on those versions.
• The 3.4.6 baseline used above carries no local patches. While chasing the remaining failures on large tables I filed #3518, #3519, #3520 and #3521; none of them depends on this patch.