listener: dequeue_proxy_listeners() livelocks at maxconn with many thread groups

Start in src/listener.c at dequeue_all_listeners() and dequeue_proxy_listeners(), then trace how listener_release() and relax_listener() interact with the queues and connection limits. Reproduce the reported stall using the provided HAProxy configuration and load steps, then check that releasing a connection returns promptly at the global and frontend limits without preventing retries for queued listeners.

Found by Claude (Anthropic) during A/B performance testing. Reproducible: 10/10 runs on master e4d1b143b. Tested patch below.

Detailed Description of the Problem

Conditions:

1. a frontend is at maxconn, or the process at global maxconn;
2. slots are freed slowly (backend serving ~10 req/s);
3. several thread groups.

A thread releasing a connection then loops in dequeue_proxy_listeners() (dequeue_all_listeners() for global maxconn), holding the PROXY lock. The watchdog reports stalls of 0.4 to 1.6 s. Sometimes it kills the worker; with the default exit-on-failure the master exits too.

Condition 3 needs no setting: on our host (KVM, 64 vCPUs exposed as 64 sockets with one L3 each), the default cpu-policy performance gives NbThreadGroups: 32.

Master e4d1b143b, counts in runs:

backend	runs	stalled	crashed
minimal uvicorn app, configuration below	3	3	2
minimal uvicorn app, test configuration run as root	7	7	1
uvicorn app with OpenTelemetry SDK	3	3	3
same, global maxconn 2000	2	2	2
same, frontend maxconn 200, 20 req/s	2	2	2
haterm (?t=60000; ?t=100 behind server maxconn 1), Python stdlib servers	11	11	0
uvicorn app with OpenTelemetry SDK, thread-groups 1	2	0	0

The stalls are systematic; whether a stall reaches the panic depends on the backend.

Expected Behavior

At maxconn, new connections wait in the accept queue. Releasing a connection takes microseconds, not seconds.

Steps to Reproduce the Behavior

Requires one CPU per thread. Tested: uvicorn 0.51.0, CPython 3.13.5, vegeta.

0. Prerequisites for the configuration below: a haproxy user and group, an empty /var/lib/haproxy; start as root.
1. Many thread groups: the default policy on such a host, or add nbthread 32 + thread-groups 32 (or 16 + 16) to global.
2. Backend at ~10 req/s: uvicorn slowapp:app --port 8000 with slowapp.py:

   import time
   async def app(scope, receive, send):
       if scope["type"] != "http":
           return
       time.sleep(0.1)  # blocks the event loop: ~10 req/s
       await send({"type": "http.response.start", "status": 200,
                   "headers": [(b"content-length", b"3")]})
       await send({"type": "http.response.body", "body": b"ok\n"})
   

3. haproxy -W -f repro.cfg (configuration below).
4. echo "GET http://127.0.0.1:10080/" | vegeta attack -rate=1000 -duration=15s -timeout=30s > /dev/null
5. Within seconds: "thread N has stopped processing traffic", trace in dequeue_proxy_listeners. Panic within about 20 s, or none; repeat if needed.

These steps, verbatim, on master:

CPUs	thread groups	runs	stalled	crashed	longest stall
64	32 (default policy)	3	3	3	1.1 s
64	32 (thread-groups 32)	3	3	3	1.1 s
32 (taskset)	32	2	2	2	1.1 s
16 (taskset)	16	3	3	2	1.1 s
8 (taskset)	32	2	2	0	0.26 s
64, patched	32	3	0	0	—

Do you have any idea what may have caused this?

listener_release() → dequeue_proxy_listeners() pops listeners until the queue is empty. relax_listener() → resume_listener() re-enables each one without checking feconn < maxconn; one slot was freed.

With shards by-group there is one listener per group. A re-enabled listener's thread accepts, finds feconn >= maxconn, and limit_proxy re-queues it. Meanwhile the releasing thread pops the next. With 32 listeners and 32 threads the queue refills faster than it drains; the loop holds the PROXY lock, and the other threads wait for it in relax_listener(). With one listener the race is rarely won.

The condition is several listeners on one frontend, each served by different threads; thread groups are one way to get them. Master with thread-groups 1 and bind :10080 shards by-thread (or 32 bind :10080 thread N) livelocks too.

History. A design flaw rather than a regression: the loop has run until empty since the listener queues (e6ca1fcd8, 1.5-dev8). Commits that change its reachability, each tested at runtime against its parent (counts in runs):

commit	release	effect	commit / parent
09b52d1c3 "MEDIUM: config: permit to start a bind on multiple groups at once"	2.8-dev8	thread-group form appears (bisect v2.7.0..2.8-dev8). The parent refuses a bind spanning 16 groups.	panics 1/1, trace in dequeue_proxy_listeners / config refused
b74336984 "MEDIUM: cpu-topo: switch to the 'performance' cpu-policy by default"	3.3-dev2	many groups by default on hosts like ours (bisect v3.2.0..v3.3.0)	3/3 stall and panic / 0/3, with the reproduction above unchanged
cb89e32f3 "MEDIUM: listeners: don't bounce listeners management between queues"	2.3-dev6	hides the single-group form (32 bind ... process 1/N)	0/3 panic / 2/3 panic in dequeue_proxy_listeners

Single-group form: 2.2.0 panics; 2.3.0 to 2.8.0 show neither panic nor elevated loop time; master stalls. Where it returned between 2.8.0 and 3.3-dev9 is not determined: before 3.1 there is no stall warning, and a livelock that does not reach the panic goes unseen.

Note for traces: in some builds dequeue_proxy_listeners() is inlined into listener_release(); the stuck frame then reads listener_release+0x... > relax_listener.

Do you have an idea how to solve the issue?

Dequeue only while there is room. The proxy task, the global queue task and set maxconn frontend already check this before calling. A listener left queued is dequeued by the next release, which decrements the counter first.

--- a/src/listener.c
+++ b/src/listener.c
@@ -757,7 +757,12 @@ void dequeue_all_listeners()
 {
 	struct listener *listener;
 
-	while ((listener = MT_LIST_POP(&global_listener_queue, struct listener *, wait_queue))) {
+	/* Same as in dequeue_proxy_listeners() below: only wake listeners up
+	 * while the process has room, or a listener woken up while it is still
+	 * full re-queues itself at once and this loop may never end.
+	 */
+	while (actconn < global.maxconn &&
+	       (listener = MT_LIST_POP(&global_listener_queue, struct listener *, wait_queue))) {
 		/* This cannot fail because the listeners are by definition in
 		 * the LI_LIMITED state.
 		 */
@@ -774,7 +779,14 @@ void dequeue_proxy_listeners(struct proxy *px, int lpx)
 {
 	struct listener *listener;
 
-	while ((listener = MT_LIST_POP(&px->listener_queue, struct listener *, wait_queue))) {
+	/* Only wake listeners up while the frontend has room. A listener woken
+	 * up while the frontend is still full re-queues itself on its next
+	 * accept, and with many listeners in the queue (e.g. one shard per
+	 * thread group) this loop could keep finding one and never return,
+	 * holding the proxy's lock until the watchdog kills the process.
+	 */
+	while (px->feconn < px->maxconn &&
+	       (listener = MT_LIST_POP(&px->listener_queue, struct listener *, wait_queue))) {
 		/* This cannot fail because the listeners are by definition in
 		 * the LI_LIMITED state.
 		 */

With the patch, master: 0 stalls, 0 crashes in 10 runs, with the limits reached in each:

• minimal uvicorn app: 3 runs;
• OpenTelemetry app at the frontend limit: 3; at the process limit: 2;
• haterm behind server maxconn 1: 2.

Side effect: a listener queued globally for another reason (e.g. fd shortage) is not retried while actconn >= global.maxconn; the next release retries it.

What is your configuration?

global
    maxconn 5000
    user haproxy
    group haproxy
    chroot /var/lib/haproxy
    stats socket /run/haproxy-repro.sock mode 600 level admin

defaults
    mode http
    maxconn 4000
    timeout connect 5s
    timeout client 50s
    timeout server 50s

frontend fe
    bind :10080
    default_backend be

backend be
    server s1 127.0.0.1:8000

haproxy -c: no warnings. No nbthread, thread-groups or cpu-policy.

Output of haproxy -vv

HAProxy version 3.5-dev8-e4d1b143b 2026/10/05 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 6.12.101+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.101-1 (2026-08-05) x86_64
Build options : 
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv -fvect-cost-model=very-cheap
  OPTIONS = USE_OPENSSL=1 USE_ZLIB=1 USE_PCRE2=1
  DEBUG   = 

Feature list : -51DEGREES +ACCEPT4 +ACME +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +FCGI +GETADDRINFO +H2 +HAVE_TCP_MD5SIG -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING +MINIDEBUG +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -PCRE +PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN -SLZ +SPOE +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY +TRACE -WURFL +ZLIB
Detected feature list : +HAVE_WORKING_TCP_MD5SIG

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=64).
Built with SSL library version : OpenSSL 3.5.6 7 Apr 2026
Running on SSL library version : OpenSSL 3.5.6 7 Apr 2026
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library FIPS mode : no
SSL library default verify directory : /usr/lib/ssl/certs
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with network namespace support.
Built with zlib version : 1.3.1
Running on zlib version : 1.3.1
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.46 2025-08-27
PCRE2 library supports JIT : no (USE_PCRE2_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 14.2.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] comp-req
	[COMP] comp-res
	[COMP] compression
	[COMP] decomp-req
	[COMP] decomp-res
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

Last Outputs and Backtraces

WARNING! thread 2 has stopped processing traffic for 108 milliseconds
WARNING! thread 26 has stopped processing traffic for 149 milliseconds
WARNING! thread 25 has stopped processing traffic for 124 milliseconds
(... 83 such warnings in this run, the longest 1060 ms ...)

PANIC! Thread 25 is about to kill the process (pid 89812).

*>Thread 25: id=0x7fea2bfff6c0 act=1 glob=0 wq=1 rq=1 tl=0 tlsz=0 rqsz=2
     25/1    loops=1090 ctxsw=1073 stuck=1 prof=0 harmless=0 isolated=0 locks=1
             cpu_ns: poll=805266645 now=1865981376 diff=1060714731
             exec_ctx: mux 'H1'
             curr_task=0x7fea20429380 (task) calls=2 last=0
               fct=0x5569f896b5b0(process_stream) ctx=(nil)
             lock_hist: U:LISTENER W:PROXY W:LISTENER U:LISTENER W:PROXY W:LISTENER U:LISTENER U:PROXY
             call trace(22):
             | 0x5569f8a4add2 <00 00 00 0f 1f 00 f3 90]: pl_wait_unlock_long.isra.0+0x42/0xb4
             | 0x5569f8a4d1f0 <89 7d c8 e8 a0 db ff ff]: relax_listener+0x540/0x552 > pl_wait_unlock_long.isra.0
             | 0x5569f8a4d4bd <d2 89 de e8 f3 f7 ff ff]: dequeue_proxy_listeners+0x2d/0x48 > relax_listener
             | 0x5569f8ade8c3 <0a 00 00 e8 4d 23 f7 ff]: session_free+0x63/0xc6a > listener_release
             | 0x5569f8adf4e5 <8b 7f 30 e8 7b f3 ff ff]: conn_session_free+0x15/0x23 > session_free
             | 0x5569f8926dde <c0 74 05 4c 89 e7 ff d0]: h1_release+0x33e/0x83f
             | 0x5569f8934c59 <48 89 df e8 47 1e ff ff]: h1s_finish_detach+0x5c9/0x700 > h1_release
             | 0x5569f8934eb8 <48 89 df e8 d8 f7 ff ff]: h1_detach+0x58/0x30a > h1s_finish_detach
             | 0x5569f8a21dbf <00 0f 29 45 c0 ff 51 58]: sc_replace_endp+0x13f/0x251
             | 0x5569f8a222c8 <8d 7d e8 e8 b8 f9 ff ff]: sc_destroy+0x18/0x11e > sc_replace_endp
             | 0x5569f896a600 <02 00 00 e8 b0 7c 0b 00]: stream_free+0x770/0x12f1 > sc_destroy
             | 0x5569f896edc5 <4c 89 e7 e8 cb b0 ff ff]: process_stream+0x3815/0x3b86 > stream_free

[ALERT]    (89808) : Current worker (89812) exited with code 134 (Aborted)
[WARNING]  (89808) : A worker process unexpectedly died and this can only be explained by a bug in haproxy or its dependencies.
[ALERT]    (89808) : exit-on-failure: killing every processes with SIGTERM
[WARNING]  (89808) : All workers exited. Exiting... (134)

Additional Information

• Filed as a regular bug: every reproduction requires a slow backend (servers are trusted, doc/internals/threat-model.txt §4). No client-only trigger found; none searched.
• dequeue_all_listeners(): same loop, seen with global maxconn 2000 < frontend maxconn 4000. Guarding only the frontend loop leaves it stalling.
• The patch applies unchanged to 2.8, 3.0, 3.2 and 3.4. With thread-groups > 1: unpatched stalls, patched none.
• Host: AMD EPYC 7542, KVM guest, Debian 13, kernel 6.12.101.
• Related, not duplicates: #2951 (listener nbconn across groups at a listener's maxconn; fixed, and noted there that 2.8 first let a listener span groups), #2011 (same call path, cause in fd_add_to_fd_list(); fixed), #2476 (rate-limit path; fixed).

🤖 Generated with Claude Code