TrainerRank oversized admission reaches CUDA OOM during Qwen3.6-35B-A3B forward

The issue is a deep memory allocation failure in the TrainerRank oversized admission path for a Qwen3.6-35B-A3B model. Start by examining the planner logic in trainer_rank/_impl.py, specifically _recover_admission_impl. Look at the memory budget calculations and the conditions that lead to admitted_oversized. The failure occurs during a forward pass, likely in MoE expert FC2 or ART's expert LoRA forward. To understand 'done', you need to determine if a smaller execution plan is possible or what additional allocation evidence is required, but the issue lacks a clear, reproducible test case.

A production TrainerRank forward failed with a CUDA OOM after the planner explicitly selected admitted_oversized. This records a confirmed failure of the opt-in oversized path for investigation. The selected estimate already exceeded the available admission budget; the evidence does not establish an underestimate or a failure of normal fit admission.

Event b02499988c7c40e98659be0a31376eb0, September 24, 2026, 2:43:31 AM MDT (08:43:31 UTC): operation=forward_micro_batches, phase=forward, decision.outcome=admitted_oversized. Actual rank fields were allow_oversized_batches=True and planner report threshold 10. The launch explicitly enabled:

ART_TRAINER_RANK_ALLOW_OVERSIZED_BATCHES=1
ART_TRAINER_RANK_PLANNER_MISS_THRESHOLD_PCT=10

The original failure explicitly says CUDA ran out of memory while allocating 2.39 GiB on a 139.81 GiB device with 2.15 GiB free. PyTorch had 133.58 GiB allocated and 243.59 MiB reserved but unallocated. These observations do not establish fragmentation.

The selected sample uses allocator native, inactive_split_bytes=0, a 3% reserve and no test memory cap. Its required estimate was carried from check ordinal 7 into selected sample ordinal 18. Planner required/peak values describe the planner observation and admission calculation; they are not absolute CUDA allocated-memory measurements. The partial peak is relative to baseline 72,455,432,192 bytes. Completed observed_peak_bytes and error_pct are both null, as intended for an interrupted OOM observation; no underestimate percentage can be calculated from them.

Runtime/model: Qwen/Qwen3.6-35B-A3B, revision 995ad96eacd98c81ed38be0c5b274b04031597b0, BF16, LoRA rank 1; Torch 2.11.0+cu128, CUDA 12.8, rank 0 / cuda:0, compute capability 9.0, device capacity 150121021440 bytes. Observed planner topology is (DP, TP, CP, PP)=(1,1,1,1), sequence parallelism false. Model geometry has 40 layers, 30 GDN layers, hidden size 2048, 256 MoE experts and top-k 8. The event identifies the policy checkpoint slot but does not record its revision. The run owner reports continuation from durable step 225 and four completed resumed callbacks; failing logical batch 230 is a FIFO inference, not a direct event field or optimizer proof.

The entire failing forward plan contains 19 requests, 1,073,093 logical tokens and 359,965 packed tokens, one slot group and four subforwards. All input/target descriptors are 1-D int64 tensors with matching lengths; gradients are enabled, logits/hidden-state outputs are disabled.

The exact failing subforward/request and allocation tensor shape are unavailable. The retained stack passes through MoE expert FC2, ART's expert LoRA forward and a TorchInductor compiled call; this locates the failure path without establishing a cause. Request descriptors are flattened in plan-group order; retained original request indices must not be applied directly to that flattened list.

The frozen run uses ART base 8ff205e543473d96557fe4b0ffbeaea19d13b420 plus a feature backport and operational instrumentation, and Caladan base 257be1824d2dd7495e6e3509fbcad9af9ee74e9a. Exact runtime source manifest SHA-256: 8ba82252431f8ce841dcbe4739d8272c925d949c20657331021a503d2eb347d0; executed ART trainer_rank/_impl.py: 6c0e9e68d32eb0067fa5206f0ff419c2840fd60922f0e90941c07b4486b1f313. In that source, _recover_admission_impl assigns admitted_oversized after exhausted memory-refusal recovery; forward tracking catches the CUDA OOM and emits the report. This is not a clean-base reproduction or a claim about current main.

Expected operational outcome was continued training with oversized admission enabled; actual outcome was a terminal CUDA OOM. The flag deliberately allows an over-budget candidate, so this issue asks whether a supported smaller execution plan can preserve the same work, and what additional allocation evidence is needed if it cannot.

Sanitized retained evidence is available: original event SHA-256 b580a842afbdbd58dab0c8f5a8bb17a455897ea62f01940ba45e98ed53623b10; safe projection d7b93efc01290300a6597894e07c954ffe0f5d27ed39e313656dadf2b769f26e. No token values, private payloads, checkpoint weights or credentials are included here. replay_complete=false (device_or_modified_input, layout_inputs_unavailable), so the retained metadata is not a complete executable GPU reproducer. No rerun or proposed fix has been tested.