Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

TrainerRank oversized admission reaches CUDA OOM during Qwen3.6-35B-A3B forward

Abierto
  • ¿Algo no está bien?
#948 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
10/100
Tipo de issue
Error
Claridad
Necesita aclaración
Estado de actividad
Activo
Stack tecnológico
python, pytorch
Área
ai-infra-agents, performance

Línea de trabajo

The issue is a deep memory allocation failure in the TrainerRank oversized admission path for a Qwen3.6-35B-A3B model. Start by examining the planner logic in trainer_rank/_impl.py, specifically _recover_admission_impl. Look at the memory budget calculations and the conditions that lead to admitted_oversized. The failure occurs during a forward pass, likely in MoE expert FC2 or ART's expert LoRA forward. To understand 'done', you need to determine if a smaller execution plan is possible or what additional allocation evidence is required, but the issue lacks a clear, reproducible test case.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

A production TrainerRank forward failed with a CUDA OOM after the planner explicitly selected admitted_oversized. This records a confirmed failure of the opt-in oversized path for investigation. The selected estimate already exceeded the available admission budget; the evidence does not establish an underestimate or a failure of normal fit admission.

Event b02499988c7c40e98659be0a31376eb0, September 24, 2026, 2:43:31 AM MDT (08:43:31 UTC): operation=forward_micro_batches, phase=forward, decision.outcome=admitted_oversized. Actual rank fields were allow_oversized_batches=True and planner report threshold 10. The launch explicitly enabled:

ART_TRAINER_RANK_ALLOW_OVERSIZED_BATCHES=1
ART_TRAINER_RANK_PLANNER_MISS_THRESHOLD_PCT=10

The original failure explicitly says CUDA ran out of memory while allocating 2.39 GiB on a 139.81 GiB device with 2.15 GiB free. PyTorch had 133.58 GiB allocated and 243.59 MiB reserved but unallocated. These observations do not establish fragmentation.

Selected admission / report field Bytes
Predicted peak 121,280,725,363
Admission required, safety factor 1.1 133,408,797,899
Available admission budget 19,621,678,285
Physical free at selected admission sample 24,125,308,928
PyTorch allocated at selected admission sample 72,457,731,584
PyTorch reserved at selected admission sample 121,867,599,872
OOM partial peak above observation baseline 73,220,957,696

The selected sample uses allocator native, inactive_split_bytes=0, a 3% reserve and no test memory cap. Its required estimate was carried from check ordinal 7 into selected sample ordinal 18. Planner required/peak values describe the planner observation and admission calculation; they are not absolute CUDA allocated-memory measurements. The partial peak is relative to baseline 72,455,432,192 bytes. Completed observed_peak_bytes and error_pct are both null, as intended for an interrupted OOM observation; no underestimate percentage can be calculated from them.

Runtime/model: Qwen/Qwen3.6-35B-A3B, revision 995ad96eacd98c81ed38be0c5b274b04031597b0, BF16, LoRA rank 1; Torch 2.11.0+cu128, CUDA 12.8, rank 0 / cuda:0, compute capability 9.0, device capacity 150121021440 bytes. Observed planner topology is (DP, TP, CP, PP)=(1,1,1,1), sequence parallelism false. Model geometry has 40 layers, 30 GDN layers, hidden size 2048, 256 MoE experts and top-k 8. The event identifies the policy checkpoint slot but does not record its revision. The run owner reports continuation from durable step 225 and four completed resumed callbacks; failing logical batch 230 is a FIFO inference, not a direct event field or optimizer proof.

The entire failing forward plan contains 19 requests, 1,073,093 logical tokens and 359,965 packed tokens, one slot group and four subforwards. All input/target descriptors are 1-D int64 tensors with matching lengths; gradients are enabled, logits/hidden-state outputs are disabled.

Subforward in plan order Input lengths Logical tokens Packed tokens GDN segments
0 76618, 80763, 84906, 89049 331336 100410 7
1 93189, 94017 187206 97683 3
2 55505, 64202, 68338, 72480 260525 83557 7
3 12430, 16565, 23973, 28111, 32251, 38974, 43111, 47238, 51373 294026 78315 17

The exact failing subforward/request and allocation tensor shape are unavailable. The retained stack passes through MoE expert FC2, ART's expert LoRA forward and a TorchInductor compiled call; this locates the failure path without establishing a cause. Request descriptors are flattened in plan-group order; retained original request indices must not be applied directly to that flattened list.

The frozen run uses ART base 8ff205e543473d96557fe4b0ffbeaea19d13b420 plus a feature backport and operational instrumentation, and Caladan base 257be1824d2dd7495e6e3509fbcad9af9ee74e9a. Exact runtime source manifest SHA-256: 8ba82252431f8ce841dcbe4739d8272c925d949c20657331021a503d2eb347d0; executed ART trainer_rank/_impl.py: 6c0e9e68d32eb0067fa5206f0ff419c2840fd60922f0e90941c07b4486b1f313. In that source, _recover_admission_impl assigns admitted_oversized after exhausted memory-refusal recovery; forward tracking catches the CUDA OOM and emits the report. This is not a clean-base reproduction or a claim about current main.

Expected operational outcome was continued training with oversized admission enabled; actual outcome was a terminal CUDA OOM. The flag deliberately allows an over-budget candidate, so this issue asks whether a supported smaller execution plan can preserve the same work, and what additional allocation evidence is needed if it cannot.

Sanitized retained evidence is available: original event SHA-256 b580a842afbdbd58dab0c8f5a8bb17a455897ea62f01940ba45e98ed53623b10; safe projection d7b93efc01290300a6597894e07c954ffe0f5d27ed39e313656dadf2b769f26e. No token values, private payloads, checkpoint weights or credentials are included here. replay_complete=false (device_or_modified_input, layout_inputs_unavailable), so the retained metadata is not a complete executable GPU reproducer. No rerun or proposed fix has been tested.

Lenguaje dominante
Python
Estrellas
10.8k
Forks
1k
Merge medio
14 h 28 min
PR fusionados (30 d)
128

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de OpenPipe/ART

Todos los issues de OpenPipe/ART

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.