Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Bug]: ollama-proxy: undocumented 120 s upstream response-header timeout turns a queued or cold-loading request into a 502

Đang mở
  • Có gì không ổn?
#57 2 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 5 ngày

@ckelseynv đang làm issue này rồi.

Từ ngày 11/9/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

Summary

ollama-proxy returns 502 {"error":"upstream error: net/http: timeout awaiting response headers"} on any upstream request that has not produced response headers within 120 s. Ollama serves one request at a time by default (OLLAMA_NUM_PARALLEL=1) and sends no headers until generation starts, so a request that sits in Ollama's queue for more than 120 s, or that waits on a model load longer than 120 s, fails through PAIR while the identical request sent directly to Ollama succeeds. The limit is not mentioned in the README, services/ollama-proxy/README.md, the flag table, or docs/troubleshooting.mdx, and there is no flag or setting that changes it. 120 s is the observed value; I did not locate the constant in the source.

PAIR version or commit

v0.1.1 release archive service-binaries-linux-x64.zip: product 0.91.7, ollama-proxy 0.26.2, nvpair-ui-broker 0.40.2, run headless via nvpair-tui 0.7.2 in tmux with NVPAIR_LOG_LEVEL=debug.

Affected component

ollama-proxy

Environment
  • Node A: Ubuntu 24.04.4, kernel 6.8.0-138, RTX 3090 24 GB, Ollama 0.30.0
  • Node B: Ubuntu 24.04.4, RTX 3060 12 GB, Ollama 0.17.5
  • Two nodes paired. Ollama was pre-existing on both and adopted by PAIR on 11434; proxy on 11435.
Steps to reproduce

1. Queue depth (single node). Ollama on node A with qwen3.6:27b Q4_K_M (about 11 s per 200-token request on the 3090). Fire 20 independent POST /v1/chat/completions concurrently at the PAIR proxy (127.0.0.1:11435), max_tokens: 200, stream: false. Fire the same 20 at Ollama directly (127.0.0.1:11434) as the control.

Path Wall clock Succeeded Failed
Direct to Ollama 226.5 s 20 0
Through PAIR 120.3 s 10 10, all 502 at 120.24 to 120.31 s

Per-request completion times through PAIR: 12.7, 23.7, 35.2, 48.4, 58.5, 70.7, 81.5, 92.8, 104.5, 115.6 s, then ten failures at 120.2 to 120.3 s. Every request whose generation would have started after 120 s was dropped. GPU utilization on the node fell to zero within one second of the 502s: Ollama cancelled the orphaned queue entries when the proxy closed the connections.

2. Same test, 8 concurrent instead of 20, so the deepest queue position starts before 120 s:

Path Wall clock Succeeded Failed
Direct to Ollama 86.5 s 8 0
Through PAIR 85.6 s 8 0

Per-request completions through PAIR: 11.4, 22.4, 32.6, 42.6, 52.9, 64.4, 75.1, 85.6 s. No 502s. The only variable between this run and the 10-of-20 failure is queue depth.

3. Cold load (cluster, model only on the peer). Node B holds deepseek-r1:14b (9 GB) on a slow disk; cold load takes 2 to 3 minutes. From node A, POST /v1/chat/completions for that model to node A's proxy. PAIR routes it to node B correctly, then 502s at 120 s. Node B's Ollama abandoned the load when the connection dropped (/api/ps empty afterwards). A retry 30 s later succeeded in 31.6 s because the file was by then in page cache.

Expected behavior

Either no fixed header timeout on inference routes, or a default at least as long as Ollama's own OLLAMA_LOAD_TIMEOUT (5 minutes) with a way to configure it (flag on ollama-proxy, settings.json, or a per-engine setting), plus a line in the proxy README and troubleshooting doc stating the default and the error text it produces.

Actual behavior

A silent 120 s limit that fails requests the engine would have served. Because the scheduler forwards immediately rather than queueing, the effective maximum burst size through PAIR is 120 s / per-request time per node, regardless of how many requests the client sends. For the 27B example that is about 10 requests per node. Workaround: keep bursts under that ceiling, or pre-load models with a direct keep_alive request before routing through PAIR.

Sanitized logs

One line per failed request, queue-depth case:

12:04:58.xxx [ollama-proxy] DEBUG proxy request complete id=... node_id=a9489f3b-... method=POST path=/v1/chat/completions target=127.0.0.1:11434 status=502 duration_ms=120002 ttfb_ms=0 err="net/http: timeout awaiting response headers"

Cold-load case:

11:00:20  request sent
11:02:20.115 [ollama-proxy] DEBUG proxy request complete id=65 node_id=70506f7d-... method=POST path=/v1/chat/completions target=192.168.1.32:11435 status=502 duration_ms=120002 ttfb_ms=0 err="net/http: timeout awaiting response headers"

Full request logs, the load generator, and GPU samples are available on request.

Ngôn ngữ chính
Go
Star
1.6k
Fork
266
Merge trung bình
3 ngày 6 giờ
Pull request đã merge (30 ngày)
22

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/Personal-AI-Router

Tất cả issue của NVIDIA/Personal-AI-Router

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.