Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

[Bug]: ollama-proxy: undocumented 120 s upstream response-header timeout turns a queued or cold-loading request into a 502

Ouverte
  • Quelque chose ne va pas ?
#57 2 commentaires 0 réactions 1 personne assignée Voir sur GitHub

Les mainteneurs répondent en général sous 5 jours

@ckelseynv y travaille déjà.

Depuis le 11/9/2026.

Évaluation

Cette issue n'a pas encore été évaluée.

Description

Summary

ollama-proxy returns 502 {"error":"upstream error: net/http: timeout awaiting response headers"} on any upstream request that has not produced response headers within 120 s. Ollama serves one request at a time by default (OLLAMA_NUM_PARALLEL=1) and sends no headers until generation starts, so a request that sits in Ollama's queue for more than 120 s, or that waits on a model load longer than 120 s, fails through PAIR while the identical request sent directly to Ollama succeeds. The limit is not mentioned in the README, services/ollama-proxy/README.md, the flag table, or docs/troubleshooting.mdx, and there is no flag or setting that changes it. 120 s is the observed value; I did not locate the constant in the source.

PAIR version or commit

v0.1.1 release archive service-binaries-linux-x64.zip: product 0.91.7, ollama-proxy 0.26.2, nvpair-ui-broker 0.40.2, run headless via nvpair-tui 0.7.2 in tmux with NVPAIR_LOG_LEVEL=debug.

Affected component

ollama-proxy

Environment
  • Node A: Ubuntu 24.04.4, kernel 6.8.0-138, RTX 3090 24 GB, Ollama 0.30.0
  • Node B: Ubuntu 24.04.4, RTX 3060 12 GB, Ollama 0.17.5
  • Two nodes paired. Ollama was pre-existing on both and adopted by PAIR on 11434; proxy on 11435.
Steps to reproduce

1. Queue depth (single node). Ollama on node A with qwen3.6:27b Q4_K_M (about 11 s per 200-token request on the 3090). Fire 20 independent POST /v1/chat/completions concurrently at the PAIR proxy (127.0.0.1:11435), max_tokens: 200, stream: false. Fire the same 20 at Ollama directly (127.0.0.1:11434) as the control.

Path Wall clock Succeeded Failed
Direct to Ollama 226.5 s 20 0
Through PAIR 120.3 s 10 10, all 502 at 120.24 to 120.31 s

Per-request completion times through PAIR: 12.7, 23.7, 35.2, 48.4, 58.5, 70.7, 81.5, 92.8, 104.5, 115.6 s, then ten failures at 120.2 to 120.3 s. Every request whose generation would have started after 120 s was dropped. GPU utilization on the node fell to zero within one second of the 502s: Ollama cancelled the orphaned queue entries when the proxy closed the connections.

2. Same test, 8 concurrent instead of 20, so the deepest queue position starts before 120 s:

Path Wall clock Succeeded Failed
Direct to Ollama 86.5 s 8 0
Through PAIR 85.6 s 8 0

Per-request completions through PAIR: 11.4, 22.4, 32.6, 42.6, 52.9, 64.4, 75.1, 85.6 s. No 502s. The only variable between this run and the 10-of-20 failure is queue depth.

3. Cold load (cluster, model only on the peer). Node B holds deepseek-r1:14b (9 GB) on a slow disk; cold load takes 2 to 3 minutes. From node A, POST /v1/chat/completions for that model to node A's proxy. PAIR routes it to node B correctly, then 502s at 120 s. Node B's Ollama abandoned the load when the connection dropped (/api/ps empty afterwards). A retry 30 s later succeeded in 31.6 s because the file was by then in page cache.

Expected behavior

Either no fixed header timeout on inference routes, or a default at least as long as Ollama's own OLLAMA_LOAD_TIMEOUT (5 minutes) with a way to configure it (flag on ollama-proxy, settings.json, or a per-engine setting), plus a line in the proxy README and troubleshooting doc stating the default and the error text it produces.

Actual behavior

A silent 120 s limit that fails requests the engine would have served. Because the scheduler forwards immediately rather than queueing, the effective maximum burst size through PAIR is 120 s / per-request time per node, regardless of how many requests the client sends. For the 27B example that is about 10 requests per node. Workaround: keep bursts under that ceiling, or pre-load models with a direct keep_alive request before routing through PAIR.

Sanitized logs

One line per failed request, queue-depth case:

12:04:58.xxx [ollama-proxy] DEBUG proxy request complete id=... node_id=a9489f3b-... method=POST path=/v1/chat/completions target=127.0.0.1:11434 status=502 duration_ms=120002 ttfb_ms=0 err="net/http: timeout awaiting response headers"

Cold-load case:

11:00:20  request sent
11:02:20.115 [ollama-proxy] DEBUG proxy request complete id=65 node_id=70506f7d-... method=POST path=/v1/chat/completions target=192.168.1.32:11435 status=502 duration_ms=120002 ttfb_ms=0 err="net/http: timeout awaiting response headers"

Full request logs, the load generator, and GPU samples are available on request.

Langage dominant
Go
Étoiles
1.6k
Forks
266
Merge moyen
3 j 6 h
PR mergées (30 j)
22

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de NVIDIA/Personal-AI-Router

Toutes les issues de NVIDIA/Personal-AI-Router

Issues similaires

Plus d'issues Go

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.