Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

GLM-5.3-Flash and full GLM-5.3 on the native CUDA engine (tracker)

Closed
  • Is something wrong?
#593 33 comments 3 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
8/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
zig
Domain
ai, machine-learning

Research direction

This is a tracker for several multi-part CUDA ports, not one task. Start by reading zig/src/cuda/nccl.zig for the existing AllReduce, AllGather and Broadcast, and the linked #587 for the quantized-weight interface in qlinear.zig. A newcomer should post a claim here on one specific piece before opening a PR, and pass the gate listed in the issue.

Written by the indexing model from the issue text.

Description

This issue collects all the GLM work on the native CUDA engine in one place. Eight threads asked overlapping questions about the same port: #509, #552, #402, #310, #308, #214, #339 and #566. Each is closed with a link here, and its proposals and receipts stay readable there. Please post claims, plans and receipts here, so two people don't build the same piece.

Where things stand

  • Native CUDA serves Nemotron 3.5 Lightning only. The native GLM family serves GLM-5.3-Flash on Metal.
  • Several working GLM-5.3-Flash CUDA ports exist outside the tree: @MiaAI-Lab's EXL3 port at TP=2 on two GB10s (their recipe, greedy replies with drafts off equal to the Python engine token for token), @Bizuayeu's TP=2 port over NCCL (#339, #548), and our own MLX affine 4-bit port, in progress.
  • Shared CUDA layers are coordinated in #548. #587 (one quantized-weight interface, qlinear.zig) is in review, and the NVFP4 entry follows it.

The gate

Every PR here passes the same gate as the rest of the engine:

  • drafted equal to "draft": false, resumed equal to fresh, concurrent equal to solo, on the device;
  • an FP64 reference for every projection class whose reduction order changes, with worst-column errors;
  • prefill and decode measured against the current release and against vLLM on the same hardware;
  • bf16 KV, and no arithmetic below today's precision;
  • drafts and concurrency through the lane core, with no serial serving path;
  • files under 600 lines, one-line comments, and no Python in the engine or its tools.
    Bit equality with the Python engine is not required.

The pieces, in order

  1. Shared layers (#548): the quantized-weight interface (#587), then NVFP4 behind it, including grouped NVFP4 experts on prompt rows.
  2. Multi-rank communication: zig/src/cuda/nccl.zig has AllReduce, AllGather and Broadcast. Send/recv and an exact reduce-scatter in rank order (#310's) come as one small PR.
  3. The GLM-5.3-Flash family on one GPU, then TP=2 and TP=3 with #310's unequal shares. Any of the ports above can be the first PR. Whichever passes the gate first lands, and the others build on it. Say here before you open one.
  4. Checkpoint formats on that family: MLX affine 4-bit, EXL3 on the reference decoder from #496, and NVIDIA's NVFP4 checkpoint read as stored, W4A16 (#308).
  5. Full GLM-5.3 (glm_moe_dsa) as its own family that reuses the Flash family's CUDA pieces rather than copying them: TP=3 (#402) and TP=4 (#552).
  6. Later, on top: the heat wait on GLM prefill under the engine's TF_HEAT_HIGH and TF_HEAT_LOW (#339), DFlash2 drafts (#509), attention on a desktop GPU with routed experts on the Sparks (#214), and Living Weights for GLM-5.3-Flash on CUDA (#566, after #591).

Answers to #552's questions

  1. Yes, a glm_moe_dsa family is wanted, sharing the Flash family's pieces as in step 5. Who is doing what is listed above.
  2. NCCL, which the CUDA engine already uses. The cluster layer's links are built for the Macs' Thunderbolt RDMA. A TF_NCCL_LIB override for hosts without a system NCCL is fine.
  3. The gate above. Bit equality with the Python engine is not required.
  4. Hand-written .cu. 1.0.3 moved Nemotron to our own CUDA kernels, and new families follow.
  5. Yes, bf16 KV. An FP8 KV cache, such as #509's fp8_ds_mla layout, would trade precision, so it isn't on the default path.
  6. Yes, a tiny synthetic checkpoint for host tests is welcome, made by a Zig tool in the tree.
Dominant language
Python
Stars
995
Forks
169
Avg merge
1d 14h
Merged PRs (30d)
281

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ashhart/TensorFold

All issues in ashhart/TensorFold

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.