Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Accuracy] PersonaPlex FP16 SwiGLU rounds SiLU before the gate product

Aperta
  • Qualcosa non va?
#1,793 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@chaofengw-nv ci sta già lavorando.

Dal 18/10/2026.

  • #1751 di @chaofengw-nv — aperta
  • #1794 di @chaofengw-nv — chiusa senza merge

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Summary

PersonaPlex FP16 SwiGLU MLPs round the activation earlier than the reference does.

  • Reference (pinned Moshi, gating.py): the torch.compile gating kernel takes the Half gate and up projections, computes
    SiLU(gate) * up in FP32, and rounds the product to Half once.
  • Family graph: graph_blocks.add_swiglu_mlp and the dual-profile _swiglu_mlp compute sigmoid(gate),
    gate * sigmoid(gate) and the product with up all in the work dtype, so each step is rounded to Half.

Minimal counterexample

The inputs are gate = 7/64 and up = 1.5, which are exact in Half and BF16.

  • Reference: Half(SiLU32(gate) * up) = 0.08648681640625.
  • Current graph: Half(Half(SiLU(gate)) * up) = 0.0865478515625.

The two results differ by one Half ULP. The FP32 product is about 600 FP32 ULPs from a Half rounding boundary, so the
outcome does not depend on the sigmoid implementation.

Evidence on real data

These results come from GB300 with TensorRT 11.2, using the reference FP16 run of the PersonaPlex audio-parity clip 7.

  • Temporal gating, layers 0/15/31, steps 0/27/28:
    • Half projections plus the FP32 pointwise product reproduce the reference gating output bitwise in 7 of 9 cases.
      The other two (layer 0 at steps 0 and 27) differ in 53 and 59 elements, cause not verified.
    • Eager Half composition differs in 1,706-2,272 of 4,096 output elements.
    • The family graph as built differs in 1,748-2,292 of 4,096.
  • Depth transformer: the same rounding is a candidate contributor to the depth-step differences under investigation.
    The full-model effect is not established yet.

Impact

Every FP16 (and BF16) SwiGLU MLP output in the PersonaPlex temporal and depth transformers is affected. FP32 builds are
not affected.

Lingua principale
Python
Stelle
276
Fork
73
Merge medio
1g 14h
PR unite (30g)
157

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/TensorRT-Model-Connect

Tutte le issue di NVIDIA/TensorRT-Model-Connect

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.