Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

AgentX warmup aborts the whole run when a request hits a keep-alive connection the server just closed

Abierto
  • ¿Algo no está bien?
#4,533 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
47/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
backend, performance

Línea de trabajo

Start by tracing the warmup request path around aiohttp_client.py and the terminal failure handling in callback_handler.py; the issue gives the failing request and log lines but no test file. Reproduce the closed keep-alive connection during a concurrent warmup, then check that a connection error raised before delivery retries once on a fresh connection and profiling proceeds.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

During an AgentX warmup, a single request fails while it is still being written: aiperf reuses a keep-alive connection that the server closed right after the previous response. aiperf treats that as a terminal warmup failure and aborts the whole benchmark (warmup_failure), so profiling never starts. Nothing was delivered to the server, so this failure is safe to retry. Retrying it once on a fresh connection would avoid aborting the benchmark.

Setup

  • InferenceX 408c015 (#3583), aiperf submodule 754356e9 (fix(agentx): allow sparse low-concurrency profile tails (#41)).
  • Scenario inferencex-agentx-mvp, public dataset semianalysis_cc_traces_weka_062126, concurrency 32. These are the recipe's own settings: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml.
  • Server: DeepSeek-V4.1-Flash, vLLM 0.32.1rc1.dev36+ge68efc685, image vllm/vllm-openai-rocm:nightly-rocm100-e68efc685ab1e6f1eda004ced7157e3ca411da2f, launched with exactly the recipe argv/env, TP4 on 4x MI355X. Client and server ran in the same pod; the client talked to 127.0.0.1.
  • The recipe pins nightly-rocm100-7f1a5398..., which is no longer on Docker Hub. e68efc685 carries the same vllm/torch/aiter versions that were in our failing environment.

What happens

The failure is always on the same request: trace 085aacf44ac5428ad3c6b4f6cd431e083832, turn 351, about 650K tokens of context and a 2.5 MB body, about 6 minutes into warmup.

Timeline from one run (profile_export.jsonl + benchmark.log):

Turn Start End Result
350 04:05:21.503 04:05:27.070 ok, on a reused keep-alive connection
351 04:05:27.072 04:05:27.072 error, sent 2 ms after turn 350 returned, failed after 0.35 ms 
04:05:27.072 ERROR    Error in aiohttp request: ClientOSError(None, 'Can not write request body for http://127.0.0.1:8888/v1/chat/completions') (aiohttp_client.py:300)
04:05:27.075 WARNING  Terminal warmup failure for trace 085aacf44ac5428ad3c6b4f6cd431e083832; aborting run early (broadcasting ProfileCancelCommand). (callback_handler.py:310)
04:05:27.076 WARNING  Run aborted (warmup_failure): A root AgentX warmup request failed, so profiling was not started. ...

profile_export records the cause as ClientConnectionResetError('Cannot write to closing transport'). In another environment the same request failed with ClientOSError(104, 'Connection reset by peer'), also 2 ms after turn 350 returned on a reused connection.

The server logged no error and stayed up in every case.

Frequency

  • Running the unchanged command in a standalone pod: 1 of 3 runs aborted this way. The other two completed warmup (354/354, 0 errors) and profiled for 10+ minutes with 0 errors.
  • In a different (sandboxed) environment with the same image and command: 3 of 3 attempts aborted on this request.
  • Replaying turn 351 by itself, either on a fresh connection or 1-2 ms after turn 350 on a single keep-alive connection, returned 200 six out of six times. So the request content is fine; the failure needs the concurrent run's timing.

Suggested fix

  • On a connection-level error raised before any request bytes reached the server (closed/closing transport, reset while writing the body), retry the request once on a new connection instead of failing it.
  • Optionally, don't make a single transport-level warmup error fatal for the whole run.

The client side of this does not depend on GPU vendor. We have only exercised it with the ROCm vLLM image above; we have not checked how often other servers close keep-alive connections at this point. Why the server closes the connection there (possibly the new HTTP frontend; the recipe sets VLLM_USE_RUST_FRONTEND=1) is a separate question that we have not verified.

Lenguaje dominante
Python
Estrellas
1.9k
Forks
343
Merge medio
2 d 40 min
PR fusionados (30 d)
431

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de SemiAnalysisAI/InferenceX

Todos los issues de SemiAnalysisAI/InferenceX

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.