Misc. bug: router mode forwards the child's Connection: close, so keep-alive clients fail with 'other side closed'

Aperta Adatta ai principianti
#32,666 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
77/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp

Direzione di ricerca

Leggi should_strip_proxy_header() in tools/server/server-models.cpp e verifica come la modalità router inoltra le intestazioni della risposta. Aggiungi test per l’intestazione Connection: close del child, se ci sono test nelle vicinanze, quindi esegui i test server pertinenti. Il lavoro è concluso quando l’intestazione di connessione del child viene rimossa dalle risposte inoltrate tramite proxy e le richieste sequenziali del client keep-alive non falliscono più.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Name and Version

llama-server built at a963f65 (Vulkan). The code is unchanged on current master (5f83714).

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server (router mode)

Command line
llama-server --models-dir /models --models-max 1 --parallel 1 --host 0.0.0.0 --port 8080
Problem description & steps to reproduce

In router mode, every proxied response carries Connection: close (from the child) next to the router's own Keep-Alive:

$ curl -s -D - -o /dev/null localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
    -d '{"model":"<any>","messages":[{"role":"user","content":"hi"}],"max_tokens":5}'
HTTP/1.1 200 OK
Connection: close
Keep-Alive: timeout=5, max=100
...

The router then closes the client connection after each response. Node's fetch/undici, which the OpenAI JS SDK and agents built on it use, reuses the pooled socket anyway. A request sent right after the previous response finishes fails with UND_ERR_SOCKET: other side closed.

Repro (Node 22), sequential streaming requests with no gap:

for (let i = 0; i < 300; i++) {
  try {
    const r = await fetch("http://127.0.0.1:8080/v1/chat/completions", {
      method: "POST", headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ model: "<any>", stream: true, max_tokens: 8,
                             messages: [{ role: "user", content: "hi" }] }) });
    await r.text();
  } catch (e) { console.log(i, e.cause?.code); }
}

Measured results:

  • Directly against the router, about 6% of requests fail when sent within 0-1 ms of the previous response; none fail at gaps of 2 ms or more.
  • Through an SSH tunnel, the close reaches the client later, so the window widens. A coding agent (Pi) hit other side closed and retried up to 38 times in one run. A local proxy that delays the close by 10 ms reproduces this: 45% of requests fail.
  • Sending Connection: close from the client removes the errors (0/300).

Cause: should_strip_proxy_header() in tools/server/server-models.cpp strips the child's keep-alive header but not its connection header. The child's Connection: close only concerns the router's own connection to the child, but it is forwarded to the client. cpp-httplib then closes the client connection (honor_connection_close), while write_response still adds Keep-Alive, since the request itself did not ask to close.

Suggested fix:

     if (header_name == "server" ||
         header_name == "transfer-encoding" ||
         header_name == "content-length" || // quick fix for https://github.com/ggml-org/llama.cpp/issues/17710
-        header_name == "keep-alive") {
+        header_name == "keep-alive" ||
+        header_name == "connection") {

I have not built and tested this patch.

Lingua principale
C++
Stelle
136k
Fork
25.7k
Merge medio
2g 1h
PR unite (30g)
524

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ggml-org/llama.cpp

Tutte le issue di ggml-org/llama.cpp

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.