Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Accuracy] PersonaPlex FP16 RMSNorm keeps FP32 weights that the reference stores in Half

Chiusa
  • Qualcosa non va?
#1,799 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@chaofengw-nv ci sta già lavorando.

Dal 18/10/2026.

  • #1751 di @chaofengw-nv — integrata
  • #1800 di @chaofengw-nv — chiusa senza merge

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

bug

Summary

PersonaPlex FP16 decoder builds use RMSNorm weights at a different precision from the reference.

  • Reference (pinned Moshi, transformer.py): the reference loads the language model with dtype=torch.float16, so every RMSNorm.alpha (BF16 in the checkpoint) is stored as Half. _rms_norm then computes in FP32 with alpha.to(var), the Half value widened to FP32.
  • Family graph: graph_ops.add_rms_norm embeds the checkpoint weight as an FP32 constant and never rounds it to the model dtype.

Most BF16 weights are exact in Half. Small ones that land in the Half subnormal range are not. In the PersonaPlex checkpoint, 5 of the 278,528 RMSNorm weights change when rounded to Half:

Weight Elements Largest relative change
transformer.layers.0.norm1.alpha 65 0.73%
transformer.layers.1.norm1.alpha 718, 968, 1453 9.1%
transformer.layers.3.norm1.alpha 2568 0.49%

For example, transformer.layers.0.norm1.alpha[65] is -1.0207295417785645e-06 in the checkpoint and -1.0132789611816406e-06 in the reference model. The reference's runtime weights for layers 0 and 31 equal the Half rounding of the checkpoint element for element.

Minimal counterexample

Inputs: x = [3, 1, 1, 1], gamma = [-1.0207295417785645e-06, 1, 1, 1] (the real BF16 checkpoint value), eps = 1e-8. The mean of x * x is exactly 3, so neither the reduction order nor the operation grouping affects the result.

  • Reference (_rms_norm on CUDA, Half weight): y[0] = -1.7285346984863281e-06.
  • Current graph (FP32 weight): y[0] = -1.7881393432617188e-06.

The two results differ by one Half subnormal step. Repeating the 4-element block to the model width 4096 gives the same outcome in all 1,024 blocks.

Evidence on real data

GB300, TensorRT 11.2.0.113, the reference FP16 run of PersonaPlex audio-parity clip 7, with temporal norm1/norm2 inputs captured at layers 0 and 31, steps 0, 27 and 28:

  • The reference CUDA _rms_norm reproduces every captured output exactly.
  • The family add_rms_norm differs in 8 elements over the 12 captures. One of them is element 65 of layer 0 norm1 at step 0, and only rounding the weight to Half removes it.
  • The other differences are associated with the FP32 operation grouping and reciprocal square root (a same-precision reordering). That is a separate numerical-alignment question, not part of this issue.

Impact

Every PersonaPlex RMSNorm in an FP16 (or BF16) build is affected. With this checkpoint, five weights in the first temporal layers change. FP32 builds are not affected. The full-model effect on the audio-parity profile is not established.

Suspected Location

families/personaplex/graph_ops.py, add_rms_norm.

Lingua principale
Python
Stelle
276
Fork
73
Merge medio
1g 7h
PR unite (30g)
166

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/TensorRT-Model-Connect

Tutte le issue di NVIDIA/TensorRT-Model-Connect

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.