Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Bug] `tilelang==0.1.11` pin in the `automodel` extra is x86-only; on aarch64 (GB200) the Automodel venv gets tilelang 0.1.12, `tile_kernels` cannot import, and DeepSeek-V4.1 training fails at the first forward

Open
  • Is something wrong?
#4,754 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

@linmuchuiyang is already working on this.

Since Oct 16, 2026.

  • #4755 by @linmuchuiyang — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch
Domain
build-system

Research direction

The fix lives in pyproject.toml: the automodel and sglang extras pin tilelang==0.1.11 only when platform_machine is x86_64. Read the markers there and the aarch64 resolution recorded in uv.lock. Done means the marker also matches aarch64 and the lock is regenerated; PR #4755 already exists, so review or verify that PR rather than starting a new change.

Written by the indexing model from the issue text.

Description

bug community-request

Describe the bug

pyproject.toml pins TileLang for the Automodel worker venv only on x86:

automodel = [
  ...
  # Automodel's FLA kernels require this TileLang version on Hopper.
  # Keep this scoped to the automodel fork; sglang intentionally pins TileLang 0.1.8.
  "tilelang==0.1.11; sys_platform == 'linux' and platform_machine == 'x86_64'",

The base dependency allows tilelang>=0.1.9,<0.1.13 on both architectures and the vllm extra pins 0.1.12, so on
aarch64 the automodel venv resolves tilelang 0.1.12 (uv.lock on main: tilelang 0.1.12 for
platform_machine != 'x86_64' and extra == 'extra-7-nemo-rl-automodel').

nemo-automodel[moe] pins tile-kernels==1.0.0, and tile_kernels/__init__.py cannot be imported on tilelang 0.1.12:

  File ".../site-packages/tile_kernels/quant/common.py", line 7, in <module>
    from tilelang.utils.target import determine_target
ModuleNotFoundError: No module named 'tilelang.utils.target'

(determine_target moved to tilelang.backend.target in 0.1.12 — deepseek-ai/TileKernels#24, open since July.)
Automodel wraps that import in safe_import_from, so _HAS_TILE_KERNELS_SINKHORN silently becomes False. For
DeepSeek-V4.1 that is fatal: deepseek_v41/model.py forces sinkhorn_backend="tilelang" whenever backend.attn == "tilelang" (the recipe setting), and the first policy forward (log-prob computation of step 1) raises

RuntimeError: dsv4 sinkhorn TileLang backend was requested, but the optional kernel is unavailable or inputs do not satisfy CUDA tensors.
  File ".../nemo_automodel/components/models/deepseek_v41/layers.py", line 114, in forward
  File ".../nemo_automodel/components/models/deepseek_v4/optimized_kernels.py", line 194, in dsv4_sinkhorn_normalize
  File ".../nemo_automodel/components/models/deepseek_v4/optimized_kernels.py", line 124, in _should_use_tilelang

The pin's own rationale ("Automodel's FLA kernels require this TileLang version") is not architecture-specific, and
Automodel main additionally caps apache-tvm-ffi<=0.1.11 because 0.1.12 "double-registers TVM FFI TypeAttrs and
SIGABRTs the tilelang kernels" — a second reason to keep the Automodel venv on the 0.1.11 stack on every architecture.
The same x86-only marker is on the sglang extra's tilelang==0.1.11 line.

Steps/Code to reproduce bug

On an aarch64 host (GB200), from a main checkout:

uv sync --frozen --extra automodel
uv run --extra automodel python - <<'PY'
import tilelang; print(tilelang.__version__)          # 0.1.12
import tile_kernels                                     # ModuleNotFoundError: tilelang.utils.target
PY

End to end: examples/configs/recipes/llm/dapo-deepseek-v4.1-12n4g-fsdp2tp1-ep48-colocated-stream-adam.yaml
(branch deepseek-v4.1-support, 12 × 4 GB200) fails in policy.get_logprobs of step 1 with the error above.
Installing tilelang==0.1.11 into the Automodel venv (uv pip install --no-deps tilelang==0.1.11) makes
tile_kernels import and the recipe trains (31 steps, three validations, on our cluster).

Expected behavior

The automodel (and sglang) extras should pin tilelang==0.1.11 on aarch64 as well:

"tilelang==0.1.11; sys_platform == 'linux' and (platform_machine == 'x86_64' or platform_machine == 'aarch64')",

(or drop the platform_machine clause entirely, as the vllm extra already does for 0.1.12), and uv.lock
regenerated. A PR with this change is attached.

Environment overview

  • NeMo RL deepseek-v4.1-support @ 8e056c2f (same pyproject lines on main @ 540d2568, 2026-10-16)
  • aarch64 GB200 NVL72, 4 GPUs per node; container derived from nvcr.io/nvidia/nemo-rl:v0.7.0 with the branch's venvs
    rebuilt by uv sync --frozen (uv 0.11.28, Python 3.13.14)
  • Automodel venv: torch 2.11.0+cu130, nemo-automodel 0.7.0+aef1fbfc, tile-kernels 1.0.0, tilelang 0.1.12 (broken) /
    0.1.11 (working)

Additional context

  • deepseek-ai/TileKernels#24 tracks the import failure; tile-kernels 2.0.0/2.1.0 (2026-09-30 / 2026-10-14) move to
    tilelang>=0.1.15 and torch>=2.13, so they are not a drop-in replacement for the 1.0.0 pin.
  • x86 Hopper users never see this because the marker matches there; the failure is aarch64-only.
Dominant language
Python
Stars
2.1k
Forks
612
Avg merge
3d 13h
Merged PRs (30d)
260

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA-NeMo/RL

All issues in NVIDIA-NeMo/RL

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.