model: Nemotron 3.5 Transcribe from safetensors on x86 cards and DGX Spark

Start with the shared placement and NVMe source attachment, which this plan says should land first for the Qwen3.8 FP4-Local/NVFP4 caller and be checked on DGX Spark. Then follow the listed Transcribe PR paths, beginning with BF16 folder service and placement; the issue names owners and expected serving behavior, but leaves multiple substantial implementation paths and maintainer questions open. Done means the specified folders and audio features serve on both the RTX 5090 host and DGX Spark, with each changed path and a real bloomery-serve session verified.

Nemotron 3.5 Transcribe를 공개된 safetensors 폴더 그대로 서빙하려는 계획입니다. 대상은 두 가지입니다: x86 호스트의 카드 한 장(카드, 호스트 RAM, NVMe를 함께 씀)과 DGX Spark(GB10, 통합 메모리)입니다. 시험 기계는 RTX 5090 32 GB x86 호스트와 DGX Spark입니다. 아래 순서로 진행합니다.

1. 목표: BF16, FP8, NVFP4 세 폴더와 DFlash2 draft를 변환 없이 읽습니다. 받아쓰기, 화자 태그, 단어 timestamp, 네이티브 MTP를 지원합니다.
2. 공용층 먼저: 폴더 배치와 NVMe tier 연결을 공용 코드에 먼저 붙입니다. 실제 호출자는 이미 붙어 있는 Qwen3.8 FP4-Local/NVFP4이고, DGX Spark에서 확인합니다.
3. 그다음 모델 작업: Transcribe PR을 하나씩 붙이고, 각 PR을 RTX 5090 x86 호스트와 DGX Spark 두 곳에서 확인합니다. Qwen3.8이 부르지 않는 공용 연산(BF16 routed, FP8 host dot)은 그 연산을 처음 쓰는 Transcribe PR에 함께 넣습니다.
4. 배치: x86 카드에서는 카드, 호스트 RAM, NVMe를 함께 씁니다. DGX Spark에서는 통합 메모리에 세 변형이 모두 올라갑니다. 매 cycle 전체를 읽는 draft와 MTP는 카드에 두고, 요청마다 한 번 쓰는 오디오 인코더와 CTC, 한 줄씩 읽는 embedding은 호스트 쪽에 둡니다.
5. 체크포인트: 아래 고정 revision의 네 폴더를 씁니다.

1. Goal and scope

I plan to implement this for two machine classes, using the existing nemotron seat and one serving slot: an x86 host with one discrete card, where the card, host RAM and NVMe tiers share the weights, and DGX Spark (GB10, 128 GB unified memory). I test on an x86 host with one RTX 5090 (32 GB VRAM, 126 GB host RAM) and on DGX Spark. The scope includes transcription, speaker tags, CTC word timestamps, native MTP2/MTP8 and optional DFlash2. The input is the released Hugging Face folders. No GGUF conversion is required.

The shared folder placement work comes first, with Qwen3.8 FP4-Local/NVFP4 as its caller on DGX Spark. Transcribe work follows the hybrid owner move in #848.

2. Checkpoint facts

Configs and safetensors headers give a 52-layer Lightning hybrid backbone: 23 Mamba-2, 23 MoE and six attention layers, width 2688, top-6 routing over 128 experts, ungated ReLU² and expert width 1856. The speech wrapper adds ASR and speaker encoders, a projection and an F32 CTC head. Input is 16 kHz mono audio, with roughly 80 ms per audio position and a 65,536-token service context.

Folder	Pinned revision	Weight payload
BF16	9d8cd6b79bfdd427b6558a5fdbf76fcaf8c75341	67.495 GB
NVFP4	539e1d17115d8b52de3b4c8b092eb55abd1cc458	21.553 GB
DFlash2	4172ee5b7bfe55f59e784922878c1bd9bdfeb6ba	1.141 GB
FP8	006d73cd8385237ed5603fc6c7e76200e50abbd2	34.667 GB

NVIDIA has published no FP4-Local folder for this model (its FP4-Local releases are Qwen3.8-Flash-Next and GLM-5.3-Flash). If one appears, it attaches through the existing Qwen3.8 FP4-Local owner. The download order is BF16, NVFP4, DFlash2, then FP8. All files at these revisions total 124,924,662,220 bytes. They will be staged before implementation. The repositories declare OpenMDW-1.1 and component notices.

FP8 uses scalar-scale E4M3 W8A8. NVFP4 is mixed: experts and attention use W4A4, while Mamba projections, ASR linears and native MTP retain FP8. Native MTP has a repeated attention/MoE block. DFlash2 has six dense layers, convolution, a candidate selector and full-context attention.

The released architecture is nemo_speechlm, with Nemotron3_5TranscribeForConditionalGeneration. A GGUF architecture field does not apply to this attach. The reference set will come from this checkpoint's published implementation and stored tensor formats.

3. Existing owners and gaps

Common owner	Attachment
Placement plan	Attaches to placement::plan_routed. Reserve caches, draft, request stage and RAM arena before assigning routed experts
Kernel dispatch	Attaches through gguf::QROWS and placement::kernels. Add compact BF16 routed execution, and FP8_B128 host dots with the K=1856 tail
Card expert leg	Attaches to the existing card/host leg and ReLU² operations
Residency	Attaches through TierBody::residency_parts and ResidencyParts; folder source ranges feed the existing arena
Second-card expert tier	Does not attach in this scope, which targets one card per host
Prompt walk	Attaches to the shared runtime walk and the hybrid owner from #848
Resident slots	Attaches at capacity one with the Nemotron recurrent stores
Serve seat	Extends the existing nemotron seat
Timing	Uses the existing serving path and timing owner

Lightning folders, Mamba-2, NVFP4 host dots, MTP rollback and DFlash2 dense/convolution/selector code already have owners. The FP8 matrix normalizer already maps scalar scales to FP8_B128; the missing work is the 64-value tail in K=1856 and host execution. The NVMe tier currently assumes file-backed gate/up/down stacks. Folder sources and Nemotron's ungated up/down parts need to attach to that tier.

Qwen3.8 can call the shared placement and NVMe work. Its released folders do not call the missing BF16 routed or hosted scalar-FP8 matrix paths. Those changes belong in common files and land with their first Transcribe caller.

4. Placement on card, host RAM and NVMe

Keep attention, Mamba, shared experts, the output head, live state and the selected draft on card. Put the input embedding in host RAM or a file mapping and gather rows. Keep the ASR/speaker encoders and CTC backed by RAM or NVMe, upload them once per request, run bounded windows and release them before decode. Use the remaining card room for routed experts, then host RAM, then the existing NVMe-backed RAM arena.

For the x86 test host (one 32 GB card), the following are conservative allocation examples [derived], in decimal GB, at context 65,536 with MTP reserved. Each includes a 4 GB peak allowance for request staging, workspaces, extra depth caches, recurrent checkpoints and allocation margin. The live census decides the actual split.

Variant	Target experts per layer, card/host	Card budget	Host weights including audio and input embedding
BF16	44/84	31.092 GB	40.928 GB
FP8	96/32	31.294 GB	9.468 GB
NVFP4	128/0	24.425 GB	1.764 GB

FP8 includes FP8_B128 scales and padding, including physical K=1920 for logical K=1856. Target f16 KV is 402.7 MB at maximum context, one MTP KV cache is 67.1 MB, and Mamba stores are 54.5 MB [derived]. Hour-long PCM, mels and projected rows take about 0.657 GB in host RAM. Windowed CTC output can be spooled; its full dense logits would take 8.591 GB [derived].

On DGX Spark the card and host share 128 GB of memory, so every variant serves whole on the card through the same planner, with no host expert leg [derived]: BF16 needs about 72.0 GB (67.495 GB weights, 0.524 GB target and MTP KV and Mamba state at context 65,536, and the 4 GB allowance), FP8 about 39.2 GB and NVFP4 about 26.1 GB.

On the x86 test host, BF16 fits the 126 GB host RAM without expert paging. On a smaller host, cold expert misses fill the shared RAM arena from local NVMe. For example, a 16 GB effective hot-expert budget, including the arena, leaves 22.554 GB of the BF16 host expert share nonresident on NVMe [derived]. Latency then depends on routing and NVMe misses. Cold audio weights and embedding pages use ordinary file-backed staging, separately from that expert arena.

Read-only draft weights still need repeated reads. Streaming the 1.141 GB DFlash2 source each cycle would cost about 163 to 380 ms from NVMe at an assumed 3 to 7 GB/s, plus upload and normalization [derived]. A cached Q8_0 image reduces bytes but still pays IO each cycle. Keeping the draft on card avoids those transfers. No decode, prefill or draft speedup is claimed before a run.

5. Shared layer PRs

• Shared folder placement and NVMe source attachment. Caller: existing Qwen3.8 FP4-Local/NVFP4. Proof machine: DGX Spark. Connect folder parts to the existing planner, arena, fills and residency, then run the changed path, the GPU gates it reaches and a real bloomery-serve session. This gives existing folder users card, RAM and NVMe placement before the Transcribe attachment.

Shared operations without a Qwen3.8 caller land in the corresponding model PR below, with common changes first. GB10 evidence does not prove AVX2 host execution.

6. Transcribe model PRs

Each item is one PR for a complete path, including its common changes, caller and serving documentation. Each runs its changed path, the GPU gates it reaches and a real bloomery-serve session on the x86 RTX 5090 host and on DGX Spark. Verification uses this checkpoint and its draft.

• BF16 folder text service and three-tier placement. Map the composite folder, add compact BF16 routed card/host operations and host embedding gathers, and attach ungated expert parts to the shared tier. User result: the BF16 folder serves on one card plus host memory, with NVMe overflow available.
• Complete audio service. Add the frontend, ASR/speaker towers, projection, CTC alignment and request-stage weight lifetime. Accept both audio_url and input_audio chat content parts through the existing media decoding owner. User result: transcription, speaker tags and word timestamps, including the supplied template's reasoning switch.
• Released FP8 and NVFP4 service. Reuse FP8_B128, add its tail/host operations, map all shards and scale aliases, and preserve each release's W8A8/W4A4 activation rules. User result: both released quantized folders serve audio on the same placement path.
• Native MTP2/MTP8. Implement Nemotron Rollback through the existing recurrent checkpoint owner and attach the released MTP blocks. User result: optional native speculation that commits Mamba state at the accepted boundary.
• DFlash2 folder. Read BF16 through Bf16Source, normalize through the existing Q8_0 loader, and reuse convolution and selector operations. For window=None, retain the full committed context plus block scratch. Apply the existing leaving-layer to entering-layer tap convention. User result: DFlash2 as an alternative to native MTP.

One reusable dump job on the x86 test host will capture intermediate states and short forced logits from the same checkpoint, with excess backbone weights in CPU RAM. It uses one short public multi-speaker clip, plus silence and malformed-input handling. The audio tower, projection and CTC fit on the card. Later format and draft work extends the same family gates. Maximum-context allocation is checked by the plan output; a second serving-context sweep is not required.

7. Open questions for the maintainer

Is there overlapping Transcribe work or an earlier attach that this should build on? I found the Lightning work in #108 and the existing draft work in #228, but no rejected Transcribe attempt in the searched issues, PRs or reachable main history.

Does the maintainer prefer a composite speech family within the existing nemotron seat? That is the proposed attachment, following #848 and the common-owner table above.

AI help

AI 코딩 에이전트가 문서를 작성한 후 이슈를 올렸습니다. 문서 리뷰와 이슈 리뷰는 제가 직접 했습니다.