[BUG] llama-perplexity crashes with CUDA illegal instruction on RTX 3090 (--kl-divergence, -fit on)
Dieses Issue hat noch niemand übernommen.
Bewertung
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Anfängerfreundlichkeit
- 25/100
Rechercherichtung
The crash is raised in ggml_cuda_compute_forward at ggml/src/ggml-cuda/ggml-cuda.cu line 2547, during a MUL_MAT op, and only with -fit on. Start by reproducing with llama-perplexity --kl-divergence and the -fit on versus -fit off flags, then compare which tensors and devices differ between the two runs. Done means the default -fit on run completes all chunks with KLD statistics printed.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Name and Version
- Version: 0.4.8 (build 12480, commit ce1a9994f)
- Built with: Clang 20.1.8 for Windows x86_64
- ggml op count: 105
- SHA256: 476353A6BB06D59578DC6E607466AC799AE74B03B1A9089A03B9912C0A185EF8
- Build date: 2026-10-14 23:34:54
Operating systems
Windows
GGML backends
CUDA
Hardware
System:
- OS: Windows 11 IoT Enterprise LTSC 10.0.26100
- CPU: AMD Ryzen 9 5950X (16C/32T)
- RAM: 64 GB
- Shell: PowerShell 5.1.26100.9711
GPU:
- Model: NVIDIA GeForce RTX 3090 (Ampere, compute capability 8.6)
- VRAM: 24576 MiB
- Driver: 617.42
- CUDA UMD Version: 13.4
Models
- Name: Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf
- Source: https://huggingface.co/localweights/Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn-GGUF/resolve/main/Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf
- Quant: IQ4_XS (imatrix), with embedded NextN/MTP head (Q8_0)
- Architecture: Qwen 3.8 Dense 27B
- Note: this GGUF contains an embedded NextN/MTP head (Q8_0). llama-perplexity does not use MTP, but mentioning it in case the loader treats MTP-embedded files differently.
Problem description & steps to reproduce
Summary
llama-perplexity.exe crashes with CUDA error: an illegal instruction was encountered when computing KL divergence (--kl-divergence + --kl-divergence-base) on Qwen3.8-27B IQ4_XS with -fit on (default). The same command with -fit off completes all 72 chunks successfully. Mainline llama.cpp (ggml-org) runs the same command to completion without errors in both modes.
Reproduction
Crashing variant (-fit on, the default — no flag needed):
.\llama-perplexity.exe `
-m "<MODEL_DIR>\Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf" `
-f "<MODEL_DIR>\wiki.test.raw" `
-fa on `
-ngl 99 `
-c 4096 `
--kl-divergence-base "<MODEL_DIR>\kld-baseline-fp16.kld" `
--kl-divergence `
--chunks 10 `
--seed 1337 `
--threads 1
Working variant (-fit off):
.\llama-perplexity.exe `
-m "<MODEL_DIR>\Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf" `
-f "<MODEL_DIR>\wiki.test.raw" `
-fa on `
-ngl 99 `
-c 4096 `
-fit off `
--kl-divergence-base "<MODEL_DIR>\kld-baseline-fp16.kld" `
--kl-divergence `
--chunks 10 `
--seed 1337 `
--threads 1
Baseline was created the same way without --kl-divergence-base/--kl-divergence.
Expected behavior
llama-perplexity --kl-divergence should complete successfully in both modes (-fit on and -fit off), as it does on ggml-org/llama.cpp.
Preliminary investigation (as requested in the template)
I ran the requested llama-completion -fit off test and both llama-perplexity variants.
Test-fit on``-fit off``llama-completion✅ PASS✅ PASSllama-perplexity --kl-divergence❌ CRASH (chunk 16)✅ PASS (all 72 chunks)
Conclusion: the crash is tied to -fit on in combination with --kl-divergence. Base inference (llama-completion) works fine in both modes.
Additional context
The same model, same --kl-divergence-base file, same arguments run to completion on ggml-org/llama.cpp (no crash, KLD statistics printed normally).
The bug is specific to BeeLlama 0.4.8 with -fit on.
Workaround: use -fit off when running llama-perplexity --kl-divergence.
I have not been able to test KVarN cache types (-ctk kvarn4 -ctv kvarn4) at all, because llama-perplexity crashes before finishing — which is the main reason I'm reporting this.
First Bad Commit
No response
Relevant log output
Crash log with -fit on (default)
Last 30 lines before crash:
10 6.1698 ± 0.1068 0.00337 ± 0.00101 0.00560 ± 0.00124 1.894 ± 0.244 % 98.764 ± 0.077 %
11 6.3279 ± 0.1055 0.00360 ± 0.00093 0.00523 ± 0.00113 1.848 ± 0.228 % 98.779 ± 0.073 %
12 6.3467 ± 0.1008 0.00318 ± 0.00086 0.00485 ± 0.00104 1.780 ± 0.217 % 98.775 ± 0.070 %
13 6.4863 ± 0.1007 0.00352 ± 0.00083 0.00486 ± 0.00097 1.836 ± 0.214 % 98.730 ± 0.069 %
14 6.6796 ± 0.1006 0.00357 ± 0.00079 0.00456 ± 0.00090 1.781 ± 0.205 % 98.730 ± 0.066 %
15 6.6054 ± 0.0956 0.00336 ± 0.00074 0.00433 ± 0.00084 1.731 ± 0.197 % 98.733 ± 0.064 %
16 6.4418 ± 0.0896 0.00324 ± 0.00071 0.00424 ± 0.00080 1.777 ± 0.196 % 98.730 ± 0.062 %
1.07.274.910 E ggml_cuda_compute_forward: MUL_MAT failed
1.07.275.095 E CUDA error: an illegal instruction was encountered
1.07.275.100 E current device: 0, in function ggml_cuda_compute_forward at ggml/src/ggml-cuda/ggml-cuda.cu:2547
1.07.275.100 E err
ggml/src/ggml-cuda/ggml-cuda.cu:113: CUDA error
Working log with -fit off
With -fit off the same command completes all 72 chunks and prints the KLD statistics block. The log ends with:
====== KL divergence statistics ======
Mean KLD: 0.008461 ± 0.000782
99.9% KLD: 1.111757
99.0% KLD: 0.012812
95.0% KLD: 0.003030
Median KLD: 0.000358
What I would like to see
llama-perplexity --kl-divergence completes successfully with -fit on (the default), so KLD benchmarks work out of the box.
Ideally also with KVarN cache types (-ctk kvarn4 -ctv kvarn4), which I haven't been able to test yet because the crash happens before finishing.
- Vorherrschende Sprache
- C++
- Sterne
- 1.2k
- Forks
- 80
- Ø Merge
- 4 T. 20 Std.
- Gemergte PRs (30 T.)
- 2
Entwicklungsumgebung
Dieses Projekt bietet weder Dev-Container noch Dockerfile noch Beitragsleitfaden – die Einrichtung liegt bei Ihnen. Beginnen Sie mit der README; die allgemeinen Schritte stehen in unserem Leitfaden für den ersten Beitrag.
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus Anbeeld/beellama.cpp
-
[BUG] KVarN assertion failure in build_inp_mem_hybrid_iswa() due to uninitialized self_kvarn_rotOffen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 68/100
Anbeeld/beellama.cpp#205 ·
-
Schwierigkeit 3/5 Ein halber Tag Anfängerfreundlichkeit 58/100
Anbeeld/beellama.cpp#206 ·
-
Schwierigkeit 3/5 1-2 Tage Anfängerfreundlichkeit 57/100
Anbeeld/beellama.cpp#204 ·
-
Schwierigkeit 4/5 3-5 Tage Anfängerfreundlichkeit 45/100
Anbeeld/beellama.cpp#203 · 2 Kommentare ·
-
Schwierigkeit 4/5 3-5 Tage Anfängerfreundlichkeit 45/100
Anbeeld/beellama.cpp#202 · 1 Kommentar ·
Alle Issues in Anbeeld/beellama.cpp
Ähnliche Issues
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
CachyOS/packageinstaller#144 ·
Maintainer antworten meist innerhalb von 1 Tag
-
documentation
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 82/100
TomerAberbach/profiler-md#951 ·
Maintainer antworten meist innerhalb von 1 Tag
-
[Bug] Default key routing (JavaStringHash) picks a different partition than the Java client for about half of keysEvtl. vergeben @0z5a hat das heute übernommen. Offen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 82/100
apache/pulsar-client-go#1575 ·
Maintainer antworten meist innerhalb von 2 Tagen
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
pocoproject/poco#5764 ·
Maintainer antworten meist innerhalb von 2 Tagen
-
bug cpp-core-guidelines low-priority
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 62/100
NatronGitHub/openfx-supportext#164 ·
Maintainer antworten meist innerhalb von 1 Tag