Fault tolerance intra-job training
@kpoeppel ci sta già lavorando.
Dal 7/7/2026.
Valutazione
Questa issue non è ancora stata valutata.
Descrizione
Fault tolerance intra-job training
During large-scale pretraining on a Slurm cluster, node failures (GPU errors, NVLink/network faults, ECC errors, etc.) are common and grow more frequent with node count and job duration, especially at node counts of 256-1024 which is our target. By default, a single node failure kills the whole job, which then has to requeue and restart from the last checkpoint which costs time. Instead, we want to allocate a small number of spare nodes alongside the training job (e.g. 2 spares). When a node fails, the job swaps in a spare and continues within the same Slurm allocation, avoiding a full requeue and minimizing lost progress. This thus works with a static world size.
Inspiration from Bloom: https://github.com/stas00/ml-engineering/tree/master/training/fault-tolerance#fault-tolerance
Tasks
- Survey existing solutions and assess fit for our stack (Megatron-LM, torchrun elastic launcher, shared sudoless HPC, Slurm). Start with NVIDIA Resiliency Extension (NVRx): https://github.com/NVIDIA/nvidia-resiliency-ext
- Check what Megatron-LM already supports natively (it integrates parts of NVRx / fault tolerance hooks)
- Evaluate torchrun elastic + rendezvous backend (e.g. c10d/etcd) for node replacement
Note any requirements that conflict with a sudoless / shared HPC environment - Design the spare-node mechanism: how spares are allocated in the Slurm job, held idle, and promoted on failure
- Implement the feature in the oellm-autoexp codebase
- Test at small scale and simulate a node failure (e.g. scancel a node, kill a process, or inject a fault) to verify recovery
- Measure recovery cost: wall-clock time to detect + swap + resume, and steps lost per failure
- Document usage (how many spares to request, config flags) for other users of the codebase
- Lingua principale
- Nessun dato sulla lingua
- Stelle
- 3
- Fork
- 0
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di OpenEuroLLM/Taskboard
-
[nucleus sparse] Experiment setupForse già presa @aaronkl l’ha presa 8 giorni fa. ApertaT4.1 - Optimization HPC T4.2 - model exploration
OpenEuroLLM/Taskboard#439 · 1 reazione · 1 assegnatario ·
-
Qwen3.5 35B-A3B hybrid MoE weak and strong scalingForse già presa @ibaucells l’ha presa 8 giorni fa. ApertaT4.1 - Optimization HPC
OpenEuroLLM/Taskboard#438 · 2 commenti · 1 assegnatario ·
-
Synthetic Generation for Multilingual Instruction TuningForse già presa @abhash-er l’ha presa 9 giorni fa. Aperta4.6 post-training
OpenEuroLLM/Taskboard#437 · 1 assegnatario ·
-
Safety and alignment evaluationForse già presa @BirgerMoell l’ha presa 9 giorni fa. Aperta4.6 post-training
OpenEuroLLM/Taskboard#436 · 1 assegnatario ·
-
Finding optimal dataset mixtures for DPOForse già presa @geoalgo l’ha presa 9 giorni fa. Aperta4.6 post-training
OpenEuroLLM/Taskboard#435 · 3 assegnatari ·