[Bug]: BitNet FFN uses SILU instead of ReLU² — perplexity 99.8 vs 17.1 on every CPU backend
@praneshnikhar arbeitet bereits daran.
Seit 09.8.2026.
Bewertung
- Schwierigkeit
- 1/5
- Geschätzter Aufwand
- Unter einer Stunde
- Anfängerfreundlichkeit
- 30/100
- Issue-Typ
- Bug
- Klarheit
- Klar beschrieben
- Aktivitätsstatus
- Veraltet
- Tech-Stack
- cpp
- Bereich
- backend, machine-learning
Rechercherichtung
Beginne mit 3rdparty/llama.cpp/src/models/bitnet.cpp und untersuche den BitNet FFN-Aktivierungsaufruf. Überprüfe anschließend ggml-org/llama.cpp#25885 und den Zustand des gepinnten Submoduls. Als erledigt gilt die Aufgabe, wenn die Korrektur aus dem Upstream in die gepinnte Abhängigkeit übernommen wurde und die gemeldeten llama-perplexity-Ergebnisse wieder in den dokumentierten Referenzbereich zurückkehren.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Model: microsoft/BitNet-b1.58-2B-4T (ggml-model-i2_s.gguf, ggml-model-tl1.gguf)
Where: 3rdparty/llama.cpp submodule → src/models/bitnet.cpp (the pinned
isHuangXin/llama.cpp@390c3077), also present in current ggml-org/llama.cpp master.
TL;DR
The BitNet FFN sub-layer applies SILU to the gate activation, but BitNet
b1.58 uses squared ReLU (relu(x)²). This is a one-line graph bug
(LLM_FFN_SILU should be LLM_FFN_RELU_SQR). It does not crash — the model
runs and emits plausible-looking text — but it is numerically mis-calibrated,
which shows up unmistakably in perplexity:
| Backend / kernel | Pure upstream (SILU) | With ReLU² fix | Official reference |
|---|---|---|---|
| x86 / I2_S (Intel) | 99.8178 ± 0.906 | 17.1086 ± 0.129 | 17.1090 ± 0.128 |
| x86 / I2_S (AMD) | 99.8178 ± 0.906 (bit-identical to Intel) | (identical by construction) | — |
| ARM / TL1 | 78.6877 ± 0.730 | 14.9641 ± 0.114 | (correct band) |
The Intel fixed value lands on the official 17.1090 to 4 decimal places
(Δ = 0.0004). The fix restores correctness on two independent kernel families
(scalar I2_S and LUT-based TL1) and across both x86 vendors, confirming this
is a graph-level bug, not a kernel/hardware issue.
Root cause
src/models/bitnet.cpp, in the per-layer FFN build:
cur = build_ffn(cur,
model.layers[il].ffn_up, NULL, model.layers[il].ffn_up_s,
model.layers[il].ffn_gate, NULL, model.layers[il].ffn_gate_s,
NULL, NULL, NULL,
NULL,
LLM_FFN_SILU, LLM_FFN_PAR, il); // ← BUG: BitNet b1.58 uses ReLU², not SILU
cb(cur, "ffn_sub_out", il);
The BitNet b1.58 architecture specifies squared ReLU in the FFN
(consistent with the reference model config). SILU here mis-scales activations
into the sub-norm, and the error compounds over 30 layers.
Fix (one line)
- LLM_FFN_SILU, LLM_FFN_PAR, il);
+ LLM_FFN_RELU_SQR, LLM_FFN_PAR, il);
LLM_FFN_RELU_SQR already exists in the enum (src/llama-graph.h), so this is
a pure activation-op swap with no new code. Throughput is unchanged (activation
op, not GEMM/LUT).
(For the origin fork Eddie-Wang1120/llama.cpp@bitnet, the same bug appears in
the pre-build_ffn monolithic form at llama.cpp:11905 as a literal
cur = ggml_silu(ctx0, cur); — replace with
cur = ggml_sqr(ctx0, ggml_relu(ctx0, cur));.)
Reproduction (clean-room, pure upstream)
Three fresh Azure VMs (Intel x86, AMD x86, ARM Ampere), pure upstream
microsoft/BitNet with recursive submodules, official 2B-4T model, standard
llama-perplexity over the same corpus:
git clone --recursive https://github.com/microsoft/BitNet.git && cd BitNet
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s # (or -q tl1 on ARM)
./build/bin/llama-perplexity -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
-f <wikitext-2-raw/wiki.test.raw> -c 2048
# -> PPL ≈ 99.82 (x86 I2_S) / ≈ 78.69 (ARM TL1) [BROKEN]
Apply the one-line fix in 3rdparty/llama.cpp/src/models/bitnet.cpp, rebuild,
re-run:
# -> PPL 17.1086 (x86 I2_S) / 14.9641 (ARM TL1) [RESTORED — matches official 17.1090]
Both broken values are far outside the correct band; both fixes land inside it.
The broken PPL differs by kernel (99.82 vs 78.69) because the mis-calibrated
activation interacts with each kernel's numerics differently — but the
brokenness itself is universal and deterministic.
Why this hasn't been pinned before
This is distinct from the existing garbage-output reports:
- Not #547 / #305 / #470 / PR #580 (missing scalar I2_S fallback on
non-AVX2 x86 → empty kernel → constantGGGG; compile-time / hardware-gated). - Not #585 / PR #469 (wrong ARM 64/16 vs 128/32 weight-group layout →
scrambled weights →!!!!; ARM-CPU-specific).
Unlike those, this bug reproduces on every backend and both x86 vendors,
does not crash or emit a constant token, and only reveals itself as a
perplexity regression (~6× worse) with plausible-but-degraded text. That is
likely why the vague quality complaints — #216 ("local model doesn't have the
same quality as online"), #348, #106, #115 — were never root-caused: nobody
measured perplexity. This bug is a strong candidate explanation for those.
Suggested routing
The fix belongs in the pinned llama.cpp (isHuangXin/llama.cpp
release-bitnet-embedding-0.6b-270m, and canonically ggml-org/llama.cpp
master). Once merged upstream, microsoft/BitNet needs only a
3rdparty/llama.cpp submodule bump. Happy to open the upstream PR (patch
verified to apply clean at the pinned commit) and link it here.
Environment
- Model:
microsoft/BitNet-b1.58-2B-4Tofficial GGUF (I2_S, TL1) - llama.cpp submodule:
isHuangXin/llama.cpp@390c3077 - Build:
355c0c4d1 (9916), clang-18, cmake - VMs: Intel x86 (AVX-512), AMD x86 (AVX-512), ARM Ampere Altra — westus3
- Cross-vendor: Intel & AMD produced bit-identical broken PPL (99.8178)
Upstream fix: the canonical fix has been opened against ggml-org/llama.cpp master: ggml-org/llama.cpp#25885 (LLM_FFN_SILU → LLM_FFN_RELU_SQR in src/models/bitnet.cpp). Recommend re-syncing the pinned llama.cpp submodule once it lands.
- Vorherrschende Sprache
- C++
- Sterne
- 40.4k
- Forks
- 3.7k
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Entwicklungsumgebung
Dieses Projekt bietet weder Dev-Container noch Dockerfile noch Beitragsleitfaden – die Einrichtung liegt bei Ihnen. Beginnen Sie mit der README; die allgemeinen Schritte stehen in unserem Leitfaden für den ersten Beitrag.
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus microsoft/BitNet
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 86/100
-
i2_s SIGSEGVs at n_ubatch >= 32: BLAS backend dequantises by a row stride 4x the real packed rowOffen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 76/100
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 86/100
-
Warning: `huggingface-cli` is deprecated and no longer works. Use `hf` instead.Evtl. vergeben @ousamabenyounes hat das vor 60 Tagen übernommen. Offen
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 92/100
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 76/100
Alle Issues in microsoft/BitNet
Ähnliche Issues
-
[request] vsg/1.1.16Offenupstream update
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 65/100
conan-io/conan-center-index#31142 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 84/100
NVIDIA/DeepStream#78 ·