[BUG] llama-perplexity crashes with CUDA illegal instruction on RTX 3090 (--kl-divergence, -fit on)

Offen
#187 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
25/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
cpp
Bereich
ai

Rechercherichtung

The crash is raised in ggml_cuda_compute_forward at ggml/src/ggml-cuda/ggml-cuda.cu line 2547, during a MUL_MAT op, and only with -fit on. Start by reproducing with llama-perplexity --kl-divergence and the -fit on versus -fit off flags, then compare which tensors and devices differ between the two runs. Done means the default -fit on run completes all chunks with KLD statistics printed.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Name and Version
  • Version: 0.4.8 (build 12480, commit ce1a9994f)
  • Built with: Clang 20.1.8 for Windows x86_64
  • ggml op count: 105
  • SHA256: 476353A6BB06D59578DC6E607466AC799AE74B03B1A9089A03B9912C0A185EF8
  • Build date: 2026-10-14 23:34:54
Operating systems

Windows

GGML backends

CUDA

Hardware

System:

  • OS: Windows 11 IoT Enterprise LTSC 10.0.26100
  • CPU: AMD Ryzen 9 5950X (16C/32T)
  • RAM: 64 GB
  • Shell: PowerShell 5.1.26100.9711

GPU:

  • Model: NVIDIA GeForce RTX 3090 (Ampere, compute capability 8.6)
  • VRAM: 24576 MiB
  • Driver: 617.42
  • CUDA UMD Version: 13.4
Models
Problem description & steps to reproduce

Summary

llama-perplexity.exe crashes with CUDA error: an illegal instruction was encountered when computing KL divergence (--kl-divergence + --kl-divergence-base) on Qwen3.8-27B IQ4_XS with -fit on (default). The same command with -fit off completes all 72 chunks successfully. Mainline llama.cpp (ggml-org) runs the same command to completion without errors in both modes.

Reproduction

Crashing variant (-fit on, the default — no flag needed):

.\llama-perplexity.exe `
    -m "<MODEL_DIR>\Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf" `
    -f "<MODEL_DIR>\wiki.test.raw" `
    -fa on `
    -ngl 99 `
    -c 4096 `
    --kl-divergence-base "<MODEL_DIR>\kld-baseline-fp16.kld" `
    --kl-divergence `
    --chunks 10 `
    --seed 1337 `
    --threads 1

Working variant (-fit off):

.\llama-perplexity.exe `
    -m "<MODEL_DIR>\Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf" `
    -f "<MODEL_DIR>\wiki.test.raw" `
    -fa on `
    -ngl 99 `
    -c 4096 `
    -fit off `
    --kl-divergence-base "<MODEL_DIR>\kld-baseline-fp16.kld" `
    --kl-divergence `
    --chunks 10 `
    --seed 1337 `
    --threads 1

Baseline was created the same way without --kl-divergence-base/--kl-divergence.

Expected behavior

llama-perplexity --kl-divergence should complete successfully in both modes (-fit on and -fit off), as it does on ggml-org/llama.cpp.

Preliminary investigation (as requested in the template)

I ran the requested llama-completion -fit off test and both llama-perplexity variants.

Test-fit on``-fit off``llama-completion✅ PASS✅ PASSllama-perplexity --kl-divergence❌ CRASH (chunk 16)✅ PASS (all 72 chunks)
Conclusion: the crash is tied to -fit on in combination with --kl-divergence. Base inference (llama-completion) works fine in both modes.

Additional context

The same model, same --kl-divergence-base file, same arguments run to completion on ggml-org/llama.cpp (no crash, KLD statistics printed normally).
The bug is specific to BeeLlama 0.4.8 with -fit on.
Workaround: use -fit off when running llama-perplexity --kl-divergence.
I have not been able to test KVarN cache types (-ctk kvarn4 -ctv kvarn4) at all, because llama-perplexity crashes before finishing — which is the main reason I'm reporting this.

First Bad Commit

No response

Relevant log output

Crash log with -fit on (default)

Last 30 lines before crash:

10 6.1698 ± 0.1068 0.00337 ± 0.00101 0.00560 ± 0.00124 1.894 ± 0.244 % 98.764 ± 0.077 %
11 6.3279 ± 0.1055 0.00360 ± 0.00093 0.00523 ± 0.00113 1.848 ± 0.228 % 98.779 ± 0.073 %
12 6.3467 ± 0.1008 0.00318 ± 0.00086 0.00485 ± 0.00104 1.780 ± 0.217 % 98.775 ± 0.070 %
13 6.4863 ± 0.1007 0.00352 ± 0.00083 0.00486 ± 0.00097 1.836 ± 0.214 % 98.730 ± 0.069 %
14 6.6796 ± 0.1006 0.00357 ± 0.00079 0.00456 ± 0.00090 1.781 ± 0.205 % 98.730 ± 0.066 %
15 6.6054 ± 0.0956 0.00336 ± 0.00074 0.00433 ± 0.00084 1.731 ± 0.197 % 98.733 ± 0.064 %
16 6.4418 ± 0.0896 0.00324 ± 0.00071 0.00424 ± 0.00080 1.777 ± 0.196 % 98.730 ± 0.062 %
1.07.274.910 E ggml_cuda_compute_forward: MUL_MAT failed
1.07.275.095 E CUDA error: an illegal instruction was encountered
1.07.275.100 E current device: 0, in function ggml_cuda_compute_forward at ggml/src/ggml-cuda/ggml-cuda.cu:2547
1.07.275.100 E err
ggml/src/ggml-cuda/ggml-cuda.cu:113: CUDA error

Working log with -fit off

With -fit off the same command completes all 72 chunks and prints the KLD statistics block. The log ends with:

====== KL divergence statistics ======
Mean KLD: 0.008461 ± 0.000782
99.9% KLD: 1.111757
99.0% KLD: 0.012812
95.0% KLD: 0.003030
Median KLD: 0.000358

What I would like to see

llama-perplexity --kl-divergence completes successfully with -fit on (the default), so KLD benchmarks work out of the box.
Ideally also with KVarN cache types (-ctk kvarn4 -ctv kvarn4), which I haven't been able to test yet because the crash happens before finishing.

Vorherrschende Sprache
C++
Sterne
1.2k
Forks
80
Ø Merge
4 T. 20 Std.
Gemergte PRs (30 T.)
2

Entwicklungsumgebung

Dieses Projekt bietet weder Dev-Container noch Dockerfile noch Beitragsleitfaden – die Einrichtung liegt bei Ihnen. Beginnen Sie mit der README; die allgemeinen Schritte stehen in unserem Leitfaden für den ersten Beitrag.

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus Anbeeld/beellama.cpp

Alle Issues in Anbeeld/beellama.cpp

Ähnliche Issues

Weitere Issues zu C++

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.