Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

AgentX warmup aborts the whole run when a request hits a keep-alive connection the server just closed

Đang mở
  • Có gì không ổn?
#4,533 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
47/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python
Lĩnh vực
backend, performance

Hướng nghiên cứu

Start by tracing the warmup request path around aiohttp_client.py and the terminal failure handling in callback_handler.py; the issue gives the failing request and log lines but no test file. Reproduce the closed keep-alive connection during a concurrent warmup, then check that a connection error raised before delivery retries once on a fresh connection and profiling proceeds.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Summary

During an AgentX warmup, a single request fails while it is still being written: aiperf reuses a keep-alive connection that the server closed right after the previous response. aiperf treats that as a terminal warmup failure and aborts the whole benchmark (warmup_failure), so profiling never starts. Nothing was delivered to the server, so this failure is safe to retry. Retrying it once on a fresh connection would avoid aborting the benchmark.

Setup

  • InferenceX 408c015 (#3583), aiperf submodule 754356e9 (fix(agentx): allow sparse low-concurrency profile tails (#41)).
  • Scenario inferencex-agentx-mvp, public dataset semianalysis_cc_traces_weka_062126, concurrency 32. These are the recipe's own settings: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml.
  • Server: DeepSeek-V4.1-Flash, vLLM 0.32.1rc1.dev36+ge68efc685, image vllm/vllm-openai-rocm:nightly-rocm100-e68efc685ab1e6f1eda004ced7157e3ca411da2f, launched with exactly the recipe argv/env, TP4 on 4x MI355X. Client and server ran in the same pod; the client talked to 127.0.0.1.
  • The recipe pins nightly-rocm100-7f1a5398..., which is no longer on Docker Hub. e68efc685 carries the same vllm/torch/aiter versions that were in our failing environment.

What happens

The failure is always on the same request: trace 085aacf44ac5428ad3c6b4f6cd431e083832, turn 351, about 650K tokens of context and a 2.5 MB body, about 6 minutes into warmup.

Timeline from one run (profile_export.jsonl + benchmark.log):

Turn Start End Result
350 04:05:21.503 04:05:27.070 ok, on a reused keep-alive connection
351 04:05:27.072 04:05:27.072 error, sent 2 ms after turn 350 returned, failed after 0.35 ms 
04:05:27.072 ERROR    Error in aiohttp request: ClientOSError(None, 'Can not write request body for http://127.0.0.1:8888/v1/chat/completions') (aiohttp_client.py:300)
04:05:27.075 WARNING  Terminal warmup failure for trace 085aacf44ac5428ad3c6b4f6cd431e083832; aborting run early (broadcasting ProfileCancelCommand). (callback_handler.py:310)
04:05:27.076 WARNING  Run aborted (warmup_failure): A root AgentX warmup request failed, so profiling was not started. ...

profile_export records the cause as ClientConnectionResetError('Cannot write to closing transport'). In another environment the same request failed with ClientOSError(104, 'Connection reset by peer'), also 2 ms after turn 350 returned on a reused connection.

The server logged no error and stayed up in every case.

Frequency

  • Running the unchanged command in a standalone pod: 1 of 3 runs aborted this way. The other two completed warmup (354/354, 0 errors) and profiled for 10+ minutes with 0 errors.
  • In a different (sandboxed) environment with the same image and command: 3 of 3 attempts aborted on this request.
  • Replaying turn 351 by itself, either on a fresh connection or 1-2 ms after turn 350 on a single keep-alive connection, returned 200 six out of six times. So the request content is fine; the failure needs the concurrent run's timing.

Suggested fix

  • On a connection-level error raised before any request bytes reached the server (closed/closing transport, reset while writing the body), retry the request once on a new connection instead of failing it.
  • Optionally, don't make a single transport-level warmup error fatal for the whole run.

The client side of this does not depend on GPU vendor. We have only exercised it with the ROCm vLLM image above; we have not checked how often other servers close keep-alive connections at this point. Why the server closes the connection there (possibly the new HTTP frontend; the recipe sets VLLM_USE_RUST_FRONTEND=1) is a separate question that we have not verified.

Ngôn ngữ chính
Python
Star
1.9k
Fork
343
Merge trung bình
2 ngày 40 phút
Pull request đã merge (30 ngày)
431

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của SemiAnalysisAI/InferenceX

Tất cả issue của SemiAnalysisAI/InferenceX

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.