dgx-spark: what is in progress on the Spark side, as a checklist

The checklist spans docs/contrib/dgx-spark.md, docs/research/tierengine.md, local-check.md, tools/contrib-check.sh, and multiple engine and GPU paths. Start by reading the linked issues and the phase documents to identify one unfinished item and its prerequisites. Done would mean completing and reporting a specific checklist item; the issue does not define a bounded task for a first contribution.

[!IMPORTANT]
By this weekend (Sunday, October 18) the port is done: every step from P1 to P5 in docs/contrib/dgx-spark.md is
finished. The goal is a working port, not more reports. After that, my Spark work moves to optimization and speed.

The decisions I need from you to finish P1 to P5 are in this comment.

This is one place for the DGX Spark work I have open or planned. Items are grouped by where they stand: in progress, waiting, found but not yet confirmed, and done. Inside each group, items sit under their phase in docs/contrib/dgx-spark.md (P1 to P5), and work on other threads comes last. It adds the planned items to what the #40 board already tracks. If you would order any of it differently, want something dropped, or already have one of these on your side, please say so here and I will adjust.

In progress

P1

• gate-engram's eviction clause on ext4 with large folios: one posix_fadvise(DONTNEED) per row leaves the edge folios resident. A fix that advises once a group is ready on a Spark; it waits on an x86_64 kernel 7.0 run before the PR.

P4

• (#46) GLM-5.3-Flash UD-Q4_K_XL and DeepSeek-V4.1-Flash Q3_K_M at serving defaults on the nvtier path, for the README rows. At serving defaults both refuse to load on one Spark before the first token (V4.1 host room 124.4 GB under the NVMe tier's floor 153.6 GB, GLM 120.4 under 142.2 on 7f3448bb). #100 is the floor fix, on top of #93 (merged as 7a7a282b): both load at serving defaults on a Spark and answer, rechecked on v0.3.2. On v0.3.2 GLM's unset residency word comes out off at serving defaults; I am measuring the cause and what it costs before I update #100. The README rows wait for #100.
• S7 (the pinned arena, device_slot and the devmap clause) and then the card leg, so a card computes on its slots: yours to me since #13, devmap included (verified on any card). Green on an RTX 5090; the DGX Spark run of the fixed version is under way, and the PR follows it. Two findings on the way are in #109.
• (8.10) The unexplained ~3.5 GB in GLM's A band: the UD-Q4_K_XL headers are read. The numbers go on #13 next.

Other threads

• (#81) Part 2 on a Zen 5 desktop (Ryzen 9 9950X): AVX-512 VNNI bodies for Q4_K's host kernels, bit-identical to AVX2, open as #86. From the same binary, prefill on Qwen3.6-35B-A3B with host experts is 6 % faster and decode is unchanged (results). A smaller fix to how the AVX2 Q4_K and Q5_K kernels load their f16 scales goes up first as its own PR, and #86 will be rebased on it. Numbers from other AVX-512 CPUs are welcome.

Waiting

P1

• The contributor gate list in local mode on a Spark, on 0.3.1 (697790c4), reported in #13: of 93 items, 29 are green, 45 red and 19 stopped, and 14 green items open a card that is neither named card. Most reds that are not Arm-only wait on inputs or gate changes on your side, and gate-gpu-e2e (the first cc 12.1 bit gate) is deferred as real-file. Four decisions are asked of you there. The rest of P1 stays open on the unpublished inputs and the first cc 12.1 bit gate.
• (Only if you want it) A page size guard: ROW_PAGE = 4096 in crates/placement/src/placement/workstation.rs is still a constant (3.1, "a sysconf read that refuses another size by name").
• (Only if you want it) sparkprep S4 (cards.sh knows two cards) and isarefuse (the CPU check is x86 only). Local and timing modes cover what the Spark needed, so I have left these.

P3

• (#76) GLM-5.3-Flash with the MTP draft on and off on a Spark: the same binary, --place a and bp, short and ~4K prompts, with the acceptance rate, for the table next to your A6000 row. Waits on #100.
• When the wider operand loads in gemm_q4k from #18, or your own router round from #17, have a build, I will time it on a Spark.
• (Only if you want it) The q5_1 lever and the router _fused regrid for Qwen3.8, each as its own PR (see #45 "What I ran into").
• (Only if you want it) The A6000 router_norm prefetch from #27, as its own PR alongside your A6000 A/B.

P4

• Row 4 of the (k) table (GLM UD-Q4_K_XL paged on one node, fills a token): S0's per-kind records (KindStats.run_ns) are on main, so this waits only on the GLM load in #100. The #46 GLM run gives serving numbers only, not this reading.

P5

• The pure placement half (host and unified into nodes[0], the 8.2 shape and the nodes_scale gate): agreed in #13, not waiting for S10. #94 moves the host and unified pool into Machine::nodes, and #95 (on top of it) adds the node plans and nodes_scale. Single-node plans match main byte for byte on a Spark. Both are rebased past 0.3.2 and pushed, ready for the box runs. stage-gpu-load-v41 needs both cards, so it waits for the box.
• The protocol half: the shared-page Link seam (item 4 in #13) is open as #96. The monotonic words (item 3) go ahead on the design in #13; the plan for the rest of P5 is #127.
• S₁ for Qwen3.8 ("S₁ is unmeasured on a Spark"): #46 has a first reading at serving defaults with the draft on, 1,038 and 1,054 tokens/s for prompts of 4,087 and 4,092 tokens on one Spark. Tell me if S₁ needs another condition (exactly P = 4096, the draft off, a lease) and I will measure that.
• N = 4: my four Spark nodes are cabled as a ring of direct links with no switch. NVIDIA documents direct links for up to three nodes, so this ring is outside that, and the switch health check and switch hop in "What N = 4 adds" cannot run here. Today's probe also opens one device a node, so three or four physical nodes cannot reach one stage on this ring without changes (#44). Tell me if a ring is worth anything to you for N > 2.

Found, not yet confirmed

P2

• (#16) The default context, where UD-Q3_K_XL got a smaller slot context than the larger UD-Q4_K_XL (2 × 18176 against 2 × 134400). The default context has moved to the card since; I have not rechecked both files on current main yet.

P4

• probe_nvread's eviction check fails on ext4 with large folios: posix_fadvise(DONTNEED) on a range that is not folio-aligned leaves the edge folios resident, so mincore still sees them. Seen on kernel 6.17 on a Spark; the reproduction is in #137, with the other problems found on the side of this work.

Other threads

• (#46) On the RTX 5090 with 64 GB, the residency host record prints a headroom of 203.7 GB on every start, on 0.3.1 and v0.3.2. I can't explain that number yet.
• (#46) With a 48 GiB limit, the RTX 5090 case where the room past the need was about 2 MB: v0.3.2's HOST_DRIFT margin on the unset climb may now turn that word off. Not run yet.
• (#86) hw_iq4xs_kernel_predicts_ik follows ik_llama.cpp's IQ4_XS AVX2 path from before its PR 2664 (3dbc2c9f). Against a dump from public ik e4bae240 it passes at 0 ULP on 64 real rows; against d4fbf593 it fails at 5 ULP on row 0. Once the reference ik moves past that commit and the IQ4_XS dump is made again, the test goes red on any machine. Details: https://github.com/midagedev/bloomery/pull/86#issuecomment-6175450213

Done

P1

• Local mode (#38), timing mode (#39), names for the Arm-only reds (#43), and the host-nightly units green in all 19 crates on a Spark after #15.
• The host-tier protocol part on a single-node Spark: rerun on b289130a (#13), 98 items, 30 green, 46 red, 22 stopped, no shared item changed its result. All four spin sites that wait on a card-written word run in passing clauses on one card (correction).
• Arm memory order of the host-tier protocol (3.1, 6): no site in the host tier is correct only under x86's ordering, so I propose no code change (#13).
• The K-quant walk on a Spark (#54): the spelled walk in #63 passes under LLVM 21, LLVM 23 and the sm_121 libNVVM cubin. Merged as 0d3c28ab.
• Fixture fill determinism on aarch64 (3.4, 6): the digest test passes on a Spark, a family fetched twice writes the same bytes, and fixture generate needs only full-length sparse sources (#13 comment).
• T1 on aarch64: #83 lets contrib-check --tier t1 run on aarch64 Linux as well as x86_64 Linux, merged as 0f81e85c. #52 has the other T1 host assumptions, found on an x86 box.

P2

• Clef-Flash, Qwen3-30B-A3B, Qwen3.6-35B-A3B and Qwen3.8 UD-Q4_K_XL answer through bloomery-serve on one Spark, and the P2 tape recorded on the Spark reads SAME 15/15 run to run (#16).

P3

• #45 Qwen3.8 decode: the host leg skipped on a layer the host serves nothing, and the hc norm chunk loaded before use. Merged as eb1ee791.
• gemv throughput in the decode step breakdown (#17; on main after #27 the gemvs read 208.8 GB/s, rerun), sm_121 against sm_86 (#42), the threads P/E split (#41), L2 row prefetch (#27), router top-k (#35).
• sparkdec §9 open term 3, the cc 12.1 JIT's registers and blocks per SM for the decode entries (#17 comment), and the first JIT cost on the Arm cores, 10.6 s with a JIT share of 6.5 s (#13).

P4

• (#77) Next-layer routing recall from the DeepSeek-V4.1-Flash decode trace made on a Spark, N = 4, 8, 16 and 32, prose and code apart (results), with the router-coverage.py extension and a self-test as #85, merged as eb186d97.
• (#71) The drive's throughput against queue depth on a Spark's drive and on an SK hynix P41 (results).
• Card read speed per memory kind and the Spark's NVMe speed (#13), cudaHostRegister cost (#33).
• Rows 1, 2, 3, 5 and 6 of the (k) table in docs/research/tierengine.md (#13), now in its section 6.
• (8.10) V4.1 hit rates ρ₁ and ρ₂ from a chat route trace recorded on a Spark (#13). Which policy 8.4's ρ should assume is asked of you there.

P5

• (#44) Follow-up to your answer to Q2: open MappedHost and the memop builders in bloomery_gpu and drop the probe's copies. Merged as #58 (b8d1ff06), and the pair rerun on that commit matches the earlier runs (#58 comment).
• 8.9 step 1 fabric numbers and step 2 protocol probe (#44), including all_gather_perf.

Other threads

• (#46) #50 one build commit variable for --version and /props (#46 request 2): merged as 0796ba0d for 0.3.1.
• (#46) #51 keep the Qwen3.8 draft on after a kept prefix (#46 request 3): option A as #89, merged at b289130a with the fixture-tier gates green.
• (#46) Request 1, the MTP save: copy only the store's first n positions, off the engine thread. #84, merged as 8505aa73; its "What I ran into" has what makes the save slow on a Spark.
• (#46) On an RTX 5090 with 64 GB of RAM: 0.3.1 loads, and leaving the churn pool out instead of raising P took decode from 54.4 to 99.0 tok/s (report, your plan). Rerun on 7f3448bb and b289130a (report): with residency unset and no limit the load could be refused. Your fix shipped in v0.3.2, and the default load there loaded in 10 of 10 starts (confirmation).
• (#48) Part 2, the Q2_0 host dot: an AVX2 row dot ported from mainline and held to its output bit for bit (#82), merged as a5d61505.
• (Docs) The rewrite of the four docs this work touches, one PR each, all merged: local-check.md #91, dgx-spark.md #101, tierengine.md #102 and sparkdec.md #103, plus the two small doc PRs #69 and #70.