Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Dedicated LocalBackend vLLM server can become unreachable during LoRA adapter reload

Aperta
  • Qualcosa non va?
#678 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
50/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
python
Ambito
ai, backend, infrastructure

Direzione di ricerca

Inizia in art/unsloth/service.py, in _reload_adapter(...), e segui la relativa chiamata da _train_dedicated; esamina il ciclo di vita del server dedicato e il percorso _training_stage in art/pipeline_trainer/trainer.py. Riproduci il fallimento del ricaricamento dell’adapter con la configurazione indicata di LocalBackend e PipelineTrainer, quindi verifica che un server vLLM temporaneamente non disponibile si riprenda o produca una diagnostica contenente il suo stato di salute e i log recenti.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

In LocalBackend dedicated mode with PipelineTrainer, some runs fail or become unhealthy immediately after an early training step when ART tries to reload the newly saved LoRA adapter into the dedicated vLLM server.

The observed failure is an httpx.ConnectError: All connection attempts failed from UnslothService._reload_adapter(...). After this starts happening, rollout workers also repeatedly fail with generic Connection error while calling /chat/completions.

This was seen after upgrading ART to main commit 48b2e5f6c384a62b44f34e1472e5fb1eeaa3474a.

Setup

  • ART commit: 48b2e5f6c384a62b44f34e1472e5fb1eeaa3474a
  • Backend: LocalBackend
  • Trainer: PipelineTrainer
  • Mode: dedicated trainer/inference GPUs
  • Hardware: 1 node, 2x H200; one GPU for trainer and one GPU for inference
  • Base model class: Llama 3.1 8B Instruct
  • Training: GRPO/CISPO via Unsloth
  • vLLM from the ART lock at this commit (vllm 0.17.0+art1 in the generated lock)
  • Checkpoint saving enabled
  • eval_at_start enabled

No private data is needed to reproduce the failure mode; this appears to be an infrastructure / adapter reload stability issue.

Symptoms

One run failed after the first training step with this stack:

Pipeline stage failed: All connection attempts failed
ExceptionGroup: unhandled errors in a TaskGroup
  File ".../art/pipeline_trainer/trainer.py", line 467, in _training_stage
    result = await self.backend.train(...)
  File ".../art/unsloth/service.py", line 628, in _train_dedicated
    await self._reload_adapter(checkpoint_dir, new_step)
  File ".../art/unsloth/service.py", line 455, in _reload_adapter
    response = await client.post(...)
  File ".../httpx/_client.py", line 1859, in post
    return await self.request(...)
httpx.ConnectError: All connection attempts failed

Other concurrent runs on the same setup remained marked as running, but stopped making useful training progress and emitted repeated rollout failures like:

Worker N: rollout failed: Connection error.
Retrying request to /chat/completions ...

Expected behavior

If the dedicated vLLM server is temporarily unavailable during adapter reload, ART should either:

  1. Wait for the server to become healthy again before resuming rollouts/training, or
  2. Restart/recover the dedicated inference server, or
  3. Fail with a clearer diagnostic that includes vLLM process health and recent server logs.

A transient adapter reload/server restart should not leave the pipeline in a state where rollout workers keep generating connection errors indefinitely.

Actual behavior

At least one run hard-failed during _reload_adapter(...) after saving a checkpoint. Other runs kept running at the Sky/job level but repeatedly failed rollout calls to the local vLLM endpoint and did not advance training meaningfully.

Possibly related

This was found while investigating forked LocalBackend/PipelineTrainer runs. There is already an open fork-specific issue here:

  • #651

The adapter reload/connectivity failure described in this issue is separate from whether the run started from the intended fork checkpoint. Even after ignoring the fork correctness problem, the dedicated vLLM server becoming unreachable during or after adapter reload appears to be a separate stability issue.

Suggested debugging additions

It would help if _reload_adapter(...) logged or surfaced:

  • Target URL/endpoint being called
  • Whether the vLLM subprocess is still alive
  • Last lines of the dedicated vLLM log file
  • Health check result before and after adapter reload
  • The checkpoint path and step being reloaded
Lingua principale
Python
Stelle
10.8k
Fork
989
Merge medio
11h 38m
PR unite (30g)
104

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di OpenPipe/ART

Tutte le issue di OpenPipe/ART

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.