Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

HTTP/WS worker threads spin forever in `__websocket_write_done()` (`sched_yield` loop) — 100% CPU with `--listen-http`

Aperta
  • Qualcosa non va?
#2,197 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@darwvin-dev ci sta già lavorando.

Dal 13/10/2026.

  • #2206 di @darwvin-dev — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
15/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
c
Ambito
backend, networking

Direzione di ricerca

Start in daemon/websocket.c: read websocket_message_push(), websocket_conn_cleanup() and __websocket_write_done() (around line 184), focusing on how wc->jobs, wc->wsi and websocket_service_lock interact. Try reproducing with --listen-http and --num-threads=2 under the concurrent curl loop, some clients aborting early, and check whether the trylock loop in __websocket_write_done() ever exits. Done means the service thread leaves lws_service() and the pool workers go idle after a client disconnects with a job still queued.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug
rtpengine version the issue has been seen with

mr14.1.1.13 (built from the git tag)

Used distribution and its version

Ubuntu 24.04 (in Docker, --net=host --privileged), libwebsockets from the Ubuntu 24.04 packages

Linux kernel version used

6.17.0-1007-aws

CPU architecture issue was seen on (see uname -m)

x86_64

Expected behaviour you didn't see

HTTP/WS worker threads finish their requests and go idle; CPU usage stays proportional to the load.

Unexpected behaviour you saw

After weeks of normal operation, rtpengine suddenly uses all CPU on the host and never recovers until restarted. We have seen it on two nodes so far; both run with listen-http enabled, used for Prometheus /metrics.

On a 2-vCPU host (so num_threads = num_cpu_cores(4) = 4):

  • Four threads of the rtpengine process, all created in the same second, each use about 50% CPU (both cores saturated, about 80% %sys). They do not own any sockets.
  • The kernel stack of all four threads, sampled repeatedly, is always sched_yield:
   do_sched_yield+0x78/0xb0
   __do_sys_sched_yield+0xe/0x20
   x64_sys_call+0x4f3/0x2680
   do_syscall_64+0x80/0xef0
   entry_SYSCALL_64_after_hwframe+0x76/0x7e
  • The only sched_yield() call in the daemon is in __websocket_write_done() (daemon/websocket.c:184 in mr14.1.1.13), so these are websocket_threads pool workers looping on mutex_trylock(&websocket_service_lock).
  • Media is not affected: 29 sessions, about 600 packets/s, almost all forwarded in the kernel. The main thread is idle in futex.

The process had been running for 35 days with about 200 open fds when it happened, so it does not look like a leak or an accumulation.

gdb backtrace of all threads attached.

rtpengine-bt-2026-10-09.txt

Steps to reproduce the problem

Not reproduced on demand yet. Our tests, based on the above:

  rtpengine ... --listen-http=127.0.0.1:8080 --num-threads=2
  # many concurrent requests, some aborted immediately
  for i in $(seq 50); do
    curl -s -o /dev/null -m 0.01 http://127.0.0.1:8080/metrics &
    curl -s -o /dev/null -H 'Connection: close' http://127.0.0.1:8080/metrics &
  done; wait
Additional program output to the terminal or logs illustrating the issue
# ps -eLo pid,tid,lstart,etime,time,pcpu,stat,comm | grep rtpengine   (busy threads only)
  3902938  686396 Thu Oct  8 20:34:19 2026  45:32 00:22:08 48.6 RLsl rtpengine
  3902938  686397 Thu Oct  8 20:34:19 2026  45:32 00:22:07 48.5 RLsl rtpengine
  3902938  686398 Thu Oct  8 20:34:19 2026  45:32 00:22:08 48.6 RLsl rtpengine
  3902938  686399 Thu Oct  8 20:34:19 2026  45:32 00:22:07 48.5 RLsl rtpengine

# mpstat -P ALL 1 5 (average)
  CPU    %usr   %sys  %soft  %steal  %idle
  all   19.43  79.56   0.67    0.00   0.34
Anything else?

Possibly related: #1520 (100% CPU with --listen-http, triggered by Connection: close requests) and #1988 (HTTP stops responding, possibly after a security scanner). The fix from #1988 (commit 1480574) is already included in the version we run.

As a workaround we are disabling listen-http and collecting statistics through the ng statistics command instead. Setting http-threads higher would presumably only make the race less likely.

Possible cause:

  1. websocket_message_push() increments wc->jobs and queues the connection in the websocket_threads pool (maximum http_threads ? : num_threads threads). The job counts as pending even though no worker is running it yet.
  2. The client of that queued request disconnects. The service thread, inside lws_service() and therefore holding websocket_service_lock, gets LWS_CALLBACK_CLOSED_HTTP → websocket_conn_cleanup(), which sets wc->wsi = NULL and blocks in cond_wait(&wc->cond, &wc->lock) until wc->jobs == 0.
  3. At that moment all pool workers are busy with other connections. Each is in __websocket_write_done(), needs websocket_service_lock to call lws_callback_on_writable(), fails the trylock, yields and retries forever.
  4. The queued job of the closed connection never gets a worker, so wc->jobs never reaches 0, so the service thread never leaves lws_service(), so websocket_service_lock is never released.

The !wc->wsi check in __websocket_write_done() protects a worker writing to the connection that is being cleaned up, but not workers writing to other connections while a job of the closing connection is still waiting in the pool queue.

Trigger conditions would be a burst of more than num_threads concurrent HTTP requests, where at least one client disconnects before its request is processed (port scanners, health checks with short timeouts, Connection: close clients, several scrapers at once). This would also explain why it happens at random after days or weeks.

The same code is unchanged in mr14.1.1.21, mr26.0.1.23, mr26.1.1.11, mr26.2.1.2 and current master.

Lingua principale
Perl
Stelle
992
Fork
459
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di sipwise/rtpengine

Tutte le issue di sipwise/rtpengine

Issue simili

Altre issue su Perl

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.