Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug] `tilelang==0.1.11` pin in the `automodel` extra is x86-only; on aarch64 (GB200) the Automodel venv gets tilelang 0.1.12, `tile_kernels` cannot import, and DeepSeek-V4.1 training fails at the first forward

Aperta
  • Qualcosa non va?
#4,754 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@linmuchuiyang ci sta già lavorando.

Dal 16/10/2026.

  • #4755 di @linmuchuiyang — aperta

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
build-system

Direzione di ricerca

The fix lives in pyproject.toml: the automodel and sglang extras pin tilelang==0.1.11 only when platform_machine is x86_64. Read the markers there and the aarch64 resolution recorded in uv.lock. Done means the marker also matches aarch64 and the lock is regenerated; PR #4755 already exists, so review or verify that PR rather than starting a new change.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug community-request

Describe the bug

pyproject.toml pins TileLang for the Automodel worker venv only on x86:

automodel = [
  ...
  # Automodel's FLA kernels require this TileLang version on Hopper.
  # Keep this scoped to the automodel fork; sglang intentionally pins TileLang 0.1.8.
  "tilelang==0.1.11; sys_platform == 'linux' and platform_machine == 'x86_64'",

The base dependency allows tilelang>=0.1.9,<0.1.13 on both architectures and the vllm extra pins 0.1.12, so on
aarch64 the automodel venv resolves tilelang 0.1.12 (uv.lock on main: tilelang 0.1.12 for
platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel').

nemo-automodel[moe] pins tile-kernels==1.0.0, and tile_kernels/__init__.py cannot be imported on tilelang 0.1.12:

  File ".../site-packages/tile_kernels/quant/common.py", line 7, in <module>
    from tilelang.utils.target import determine_target
ModuleNotFoundError: No module named 'tilelang.utils.target'

(determine_target moved to tilelang.backend.target in 0.1.12 — deepseek-ai/TileKernels#24, open since July.)
Automodel wraps that import in safe_import_from, so _HAS_TILE_KERNELS_SINKHORN silently becomes False. For
DeepSeek-V4.1 that is fatal: deepseek_v41/model.py forces sinkhorn_backend="tilelang" whenever backend.attn == "tilelang" (the recipe setting), and the first policy forward (log-prob computation of step 1) raises

RuntimeError: dsv4 sinkhorn TileLang backend was requested, but the optional kernel is unavailable or inputs do not satisfy CUDA tensors.
  File ".../nemo_automodel/components/models/deepseek_v41/layers.py", line 114, in forward
  File ".../nemo_automodel/components/models/deepseek_v4/optimized_kernels.py", line 194, in dsv4_sinkhorn_normalize
  File ".../nemo_automodel/components/models/deepseek_v4/optimized_kernels.py", line 124, in _should_use_tilelang

The pin's own rationale ("Automodel's FLA kernels require this TileLang version") is not architecture-specific, and
Automodel main additionally caps apache-tvm-ffi<=0.1.11 because 0.1.12 "double-registers TVM FFI TypeAttrs and
SIGABRTs the tilelang kernels" — a second reason to keep the Automodel venv on the 0.1.11 stack on every architecture.
The same x86-only marker is on the sglang extra's tilelang==0.1.11 line.

Steps/Code to reproduce bug

On an aarch64 host (GB200), from a main checkout:

uv sync --frozen --extra automodel
uv run --extra automodel python - <<'PY'
import tilelang; print(tilelang.__version__)          # 0.1.12
import tile_kernels                                     # ModuleNotFoundError: tilelang.utils.target
PY

End to end: examples/configs/recipes/llm/dapo-deepseek-v4.1-12n4g-fsdp2tp1-ep48-colocated-stream-adam.yaml
(branch deepseek-v4.1-support, 12 × 4 GB200) fails in policy.get_logprobs of step 1 with the error above.
Installing tilelang==0.1.11 into the Automodel venv (uv pip install --no-deps tilelang==0.1.11) makes
tile_kernels import and the recipe trains (31 steps, three validations, on our cluster).

Expected behavior

The automodel (and sglang) extras should pin tilelang==0.1.11 on aarch64 as well:

"tilelang==0.1.11; sys_platform == 'linux' and (platform_machine == 'x86_64' or platform_machine == 'aarch64')",

(or drop the platform_machine clause entirely, as the vllm extra already does for 0.1.12), and uv.lock
regenerated. A PR with this change is attached.

Environment overview

  • NeMo RL deepseek-v4.1-support @ 8e056c2f (same pyproject lines on main @ 540d2568, 2026-10-16)
  • aarch64 GB200 NVL72, 4 GPUs per node; container derived from nvcr.io/nvidia/nemo-rl:v0.7.0 with the branch's venvs
    rebuilt by uv sync --frozen (uv 0.11.28, Python 3.13.14)
  • Automodel venv: torch 2.11.0+cu130, nemo-automodel 0.7.0+aef1fbfc, tile-kernels 1.0.0, tilelang 0.1.12 (broken) /
    0.1.11 (working)

Additional context

  • deepseek-ai/TileKernels#24 tracks the import failure; tile-kernels 2.0.0/2.1.0 (2026-09-30 / 2026-10-14) move to
    tilelang>=0.1.15 and torch>=2.13, so they are not a drop-in replacement for the 1.0.0 pin.
  • x86 Hopper users never see this because the marker matches there; the failure is aarch64-only.
Lingua principale
Python
Stelle
2.1k
Fork
612
Merge medio
3g 14h
PR unite (30g)
267

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA-NeMo/RL

Tutte le issue di NVIDIA-NeMo/RL

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.