Peers: releasing a superseded connection clears the replacement alive flag

Comienza en src/peers.c, en peer_session_release(), e inspecciona su comprobación de propiedad junto con __peer_session_deinit(). Compila HAProxy y ejecuta repro-collision.py contra él; se considera hecho cuando el peer de reemplazo permanece en RUNNING durante la ventana de observación de 2,5 segundos en todas las pruebas indicadas.

Detailed Description of the Problem

After a duplicate incoming connection replaces an established peer connection, peer_session_release() for the superseded applet clears PEER_F_ALIVE without checking that the applet still owns the peer. The ownership check only guards __peer_session_deinit(). The collision backoff is 50 to 2049 ms, while the first heartbeat is due after 3000 ms, so the replacement can be declared dead before its first heartbeat is due.

A synthetic client completes two successive handshakes as the same peer, closes the old socket, then watches the replacement for 2.5 seconds. Unmodified 3.4.6 fails five of five trials, as do builds carrying the #3517 patch with and without the #3518 patch (neither touches peer_session_release()). A control that performs only one handshake survives five of five on unmodified 3.4.6, so the replacement is the trigger. With the ownership guard below, five of five trials survive.

Expected Behavior

Releasing a superseded applet should not change the liveness state of the replacement connection. A newly established peer must have time to send its first heartbeat.

Steps to Reproduce the Behavior

Save the following as repro-collision.py and run:

python3 repro-collision.py /absolute/path/to/haproxy

This starts one HAProxy process on loopback with a small table, no TLS and no reload, and terminates it afterward. Python 3.13 was used. The program prints SURVIVED or PREMATURE_CLOSE for each trial.

import sys,os,signal,socket,subprocess,time,tempfile
from pathlib import Path
def command(path: Path, request: str) -> str:
    """Send a runtime request to path and return its reply; raise on I/O errors."""
    with socket.socket(socket.AF_UNIX) as client:
        client.settimeout(2)
        client.connect(str(path))
        client.sendall((request + "\n").encode())
        chunks: list[bytes] = []
        while chunk := client.recv(65536):
            chunks.append(chunk)
        return b"".join(chunks).decode()

def unused_port() -> int:
    """Return an available loopback TCP port, released for HAProxy to bind."""
    with socket.socket() as listener:
        listener.bind(("127.0.0.1", 0))
        return int(listener.getsockname()[1])
from types import SimpleNamespace
h=SimpleNamespace(command=command, unused_port=unused_port)
binary=Path(sys.argv[1]).resolve()
root=Path(tempfile.mkdtemp(prefix='collision-'))
port=h.unused_port(); remote=h.unused_port()
(root/'haproxy.cfg').write_text(f'''global
  localpeer local
  nbthread 1
  stats socket {root}/admin.sock level admin
defaults
  mode tcp
  timeout connect 1s
  timeout client 10s
  timeout server 10s
peers shared
  peer local 127.0.0.1:{port}
  peer remote 127.0.0.1:{remote}
  table marker type ip size 100 expire 1h store gpc0
''')
log=(root/'haproxy.log').open('w'); connections=[]
p=subprocess.Popen([str(binary),'-db','-f',str(root/'haproxy.cfg')],stdout=log,stderr=log)
try:
 time.sleep(6)
 for trial in range(5):
  for phase in ['first','replace']:
   s=socket.create_connection(('127.0.0.1',port)); connections.append(s)
   s.settimeout(2); s.sendall(b'HAProxyS 2.1\nlocal\nremote 123 1\n')
   print(trial,phase,'handshake',repr(s.recv(4096)),flush=True)
   if phase=='replace':connections[-2].close()
   time.sleep(.1)
   print(h.command(root/'admin.sock','show peers'),flush=True)
  started=time.monotonic()
  while time.monotonic()-started<2.5:
   time.sleep(.1)
   state=h.command(root/'admin.sock','show peers')
   line=next(l for l in state.splitlines() if 'id=remote(' in l)
   if 'app_state=RUNNING' not in line:
    print('PREMATURE_CLOSE',trial,round(time.monotonic()-started,2),state,flush=True);break
  else:print('SURVIVED',trial,flush=True)
  for s in connections:s.close()
  connections=[];time.sleep(.2)
finally:
 for s in connections:s.close()
 p.terminate();p.wait(timeout=5)

Do you have any idea what may have caused this?

When the second handshake arrives, peer_session_forceshutdown() is called on the established applet (line 3121 in 3.4.6). It runs __peer_session_deinit(), which sets peer->appctx = NULL (line 1107), and the handshake then installs the new applet and sets PEER_F_ALIVE for it (lines 3134 and 3135). The old applet's svcctx still points at the peer. When that applet is finally released, peer->appctx != appctx, so __peer_session_deinit() is correctly skipped, but the PEER_F_ALIVE clear on line 1173 runs unconditionally and erases the replacement's flag. When the 50 to 2049 ms collision reconnect expires, the sync task finds the flag clear and force-closes the replacement, before any received message could have set it again (the first heartbeat is only due at 3 s).

Do you have an idea how to solve the issue?

--- a/src/peers.c
+++ b/src/peers.c
@@ -1168,9 +1168,10 @@
 	/* peer session identified */
 	if (peer) {
 		HA_SPIN_LOCK(PEER_LOCK, &peer->lock);
-		if (peer->appctx == appctx)
+		if (peer->appctx == appctx) {
 			__peer_session_deinit(peer);
-		peer->flags &= ~PEER_F_ALIVE;
+			peer->flags &= ~PEER_F_ALIVE;
+		}
 		HA_SPIN_UNLOCK(PEER_LOCK, &peer->lock);
 		TRACE_STATE("peer session released", PEERS_EV_SESS_END, appctx, peer);
 	}

Note on placement. After peer_session_forceshutdown(), __peer_session_deinit() has already set peer->appctx = NULL, so with this change the release of a force-closed applet no longer clears PEER_F_ALIVE on any path, not only the collision one. I found no behavioural difference in 3.4.6: the collision path sets PEER_F_ALIVE for the replacement right after the shutdown, the unresponsive-peer path has already cleared it before shutting down, and the stopping path does not reconnect remote peers. If you would rather keep the flag cleared on every teardown, moving the clear into __peer_session_deinit() and dropping it from peer_session_release() does that; in the collision path deinit runs before the replacement sets the flag, so the order stays correct. I have only tested the form above.

Scope and validation

With this patch and the #3520 patch applied together, an isolated three-node simultaneous-reload test showed no heartbeat timeouts or reconnects after initial establishment settled, in two 35-second windows per run, with plain TCP and one thread and again with mutual TLS and eight threads (peer state sampled every 200 ms). The #3517 two-section reload reproducer and the #3518 100,001-entry mutual-TLS resync reproducer with eight threads also pass with all patches applied. A search of upstream issues did not find an exact match. The nearby #3234 was a different shutdown-handling defect in the peer applet, where a peer listener stopped responding.

What is your configuration?

The reproducer substitutes temporary socket paths and available loopback ports in this configuration:

global
  localpeer local
  nbthread 1
  stats socket {root}/admin.sock level admin
defaults
  mode tcp
  timeout connect 1s
  timeout client 10s
  timeout server 10s
peers shared
  peer local 127.0.0.1:{port}
  peer remote 127.0.0.1:{remote}
  table marker type ip size 100 expire 1h store gpc0

Output of haproxy -vv

Unmodified 3.4.6, which fails five of five, built with make TARGET=linux-glibc USE_OPENSSL= USE_LUA= USE_PCRE2=:

HAProxy version 3.4.6-56332c5 2026/09/28 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2031.
Known bugs: http://www.haproxy.org/bugs/bugs-3.4.6.html
Running on: Linux 6.12.107+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.107-1 (2026-08-29) x86_64
Build options : 
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv -fvect-cost-model=very-cheap
  OPTIONS = USE_OPENSSL= USE_LUA= USE_PCRE2=
  DEBUG   = 

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO +HAVE_TCP_MD5SIG -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER -OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ -SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY +TRACE -WURFL -ZLIB
Detected feature list : +HAVE_WORKING_TCP_MD5SIG

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=8).
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 14.2.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] comp-req
	[COMP] comp-res
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

The patched control builds were the same source with the #3517 patch applied, with and without #3518, built with AWS-LC, QUIC, Lua and PCRE2:

HAProxy version 3.4.6-56332c5-peers-reload1 2026/09/28 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2031.
Known bugs: http://www.haproxy.org/bugs/bugs-3.4.6.html
Running on: Linux 6.12.107+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.107-1 (2026-08-29) x86_64
Build options : 
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv -fvect-cost-model=very-cheap -fstack-protector-strong -D_FORTIFY_SOURCE=3 -fPIE
  OPTIONS = USE_OPENSSL_AWSLC=1 USE_LUA=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_STRICT_ACTION

Feature list : -51DEGREES +ACCEPT4 +ACME +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO +HAVE_TCP_MD5SIG -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL +OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY +TRACE -WURFL -ZLIB
Detected feature list : +HAVE_WORKING_TCP_MD5SIG

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=8).
Built with SSL library version : OpenSSL 1.1.1 (compatible; AWS-LC 5.11.0)
Running on SSL library version : AWS-LC 5.11.0
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library FIPS mode : no
SSL library default verify directory : /etc/ssl/certs
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
QUIC: connection sock-per-conn mode support : yes
QUIC: GSO emission support : yes
Built with Lua version : Lua 5.4.7
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.46 2025-08-27
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 14.2.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       qmux : mode=HTTP  side=FE|BE  mux=QMUX  flags=HTX|NO_UPG
       quic : mode=HTTP  side=FE|BE  mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] comp-req
	[COMP] comp-res
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

Last Outputs and Backtraces

No crash. Selected control output:

PREMATURE_CLOSE 0 1.21 0x55ef31f683f0: [19/Feb/1970:12:02:26] id=shared disabled=0 flags=0x300003 resync_timeout=<PAST> task_calls=7
PREMATURE_CLOSE 1 0.2 0x55ef31f683f0: [19/Feb/1970:12:02:26] id=shared disabled=0 flags=0x300003 resync_timeout=<PAST> task_calls=11
PREMATURE_CLOSE 2 1.26 0x55ef31f683f0: [19/Feb/1970:12:02:26] id=shared disabled=0 flags=0x300003 resync_timeout=<PAST> task_calls=16
PREMATURE_CLOSE 3 1.62 0x55ef31f683f0: [19/Feb/1970:12:02:26] id=shared disabled=0 flags=0x300003 resync_timeout=<PAST> task_calls=20
PREMATURE_CLOSE 4 1.51 0x55ef31f683f0: [19/Feb/1970:12:02:26] id=shared disabled=0 flags=0x300003 resync_timeout=<PAST> task_calls=24

Additional Information

Field results: a 3.4.6 build carrying this patch together with the fixes proposed in #3517, #3518, #3520 and #3521 was deployed to a four-node production cluster on 2026-10-02. Each node has two peers sections, the nodes talk over mTLS, and the replicated tables hold 1.3 to 2.4 million entries each. After deployment, reloading the nodes one at a time and then all four at once preserved every checked entry and test marker. The local handovers took 15 to 33 seconds and the new worker sent three to six progress heartbeats (#3521) during each.