[BUG] llama-perplexity crashes with CUDA illegal instruction on RTX 3090 (--kl-divergence, -fit on)

Aperta
#187 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
ai

Direzione di ricerca

The crash is raised in ggml_cuda_compute_forward at ggml/src/ggml-cuda/ggml-cuda.cu line 2547, during a MUL_MAT op, and only with -fit on. Start by reproducing with llama-perplexity --kl-divergence and the -fit on versus -fit off flags, then compare which tensors and devices differ between the two runs. Done means the default -fit on run completes all chunks with KLD statistics printed.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Name and Version
  • Version: 0.4.8 (build 12480, commit ce1a9994f)
  • Built with: Clang 20.1.8 for Windows x86_64
  • ggml op count: 105
  • SHA256: 476353A6BB06D59578DC6E607466AC799AE74B03B1A9089A03B9912C0A185EF8
  • Build date: 2026-10-14 23:34:54
Operating systems

Windows

GGML backends

CUDA

Hardware

System:

  • OS: Windows 11 IoT Enterprise LTSC 10.0.26100
  • CPU: AMD Ryzen 9 5950X (16C/32T)
  • RAM: 64 GB
  • Shell: PowerShell 5.1.26100.9711

GPU:

  • Model: NVIDIA GeForce RTX 3090 (Ampere, compute capability 8.6)
  • VRAM: 24576 MiB
  • Driver: 617.42
  • CUDA UMD Version: 13.4
Models
Problem description & steps to reproduce

Summary

llama-perplexity.exe crashes with CUDA error: an illegal instruction was encountered when computing KL divergence (--kl-divergence + --kl-divergence-base) on Qwen3.8-27B IQ4_XS with -fit on (default). The same command with -fit off completes all 72 chunks successfully. Mainline llama.cpp (ggml-org) runs the same command to completion without errors in both modes.

Reproduction

Crashing variant (-fit on, the default — no flag needed):

.\llama-perplexity.exe `
    -m "<MODEL_DIR>\Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf" `
    -f "<MODEL_DIR>\wiki.test.raw" `
    -fa on `
    -ngl 99 `
    -c 4096 `
    --kl-divergence-base "<MODEL_DIR>\kld-baseline-fp16.kld" `
    --kl-divergence `
    --chunks 10 `
    --seed 1337 `
    --threads 1

Working variant (-fit off):

.\llama-perplexity.exe `
    -m "<MODEL_DIR>\Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf" `
    -f "<MODEL_DIR>\wiki.test.raw" `
    -fa on `
    -ngl 99 `
    -c 4096 `
    -fit off `
    --kl-divergence-base "<MODEL_DIR>\kld-baseline-fp16.kld" `
    --kl-divergence `
    --chunks 10 `
    --seed 1337 `
    --threads 1

Baseline was created the same way without --kl-divergence-base/--kl-divergence.

Expected behavior

llama-perplexity --kl-divergence should complete successfully in both modes (-fit on and -fit off), as it does on ggml-org/llama.cpp.

Preliminary investigation (as requested in the template)

I ran the requested llama-completion -fit off test and both llama-perplexity variants.

Test-fit on``-fit off``llama-completion✅ PASS✅ PASSllama-perplexity --kl-divergence❌ CRASH (chunk 16)✅ PASS (all 72 chunks)
Conclusion: the crash is tied to -fit on in combination with --kl-divergence. Base inference (llama-completion) works fine in both modes.

Additional context

The same model, same --kl-divergence-base file, same arguments run to completion on ggml-org/llama.cpp (no crash, KLD statistics printed normally).
The bug is specific to BeeLlama 0.4.8 with -fit on.
Workaround: use -fit off when running llama-perplexity --kl-divergence.
I have not been able to test KVarN cache types (-ctk kvarn4 -ctv kvarn4) at all, because llama-perplexity crashes before finishing — which is the main reason I'm reporting this.

First Bad Commit

No response

Relevant log output

Crash log with -fit on (default)

Last 30 lines before crash:

10 6.1698 ± 0.1068 0.00337 ± 0.00101 0.00560 ± 0.00124 1.894 ± 0.244 % 98.764 ± 0.077 %
11 6.3279 ± 0.1055 0.00360 ± 0.00093 0.00523 ± 0.00113 1.848 ± 0.228 % 98.779 ± 0.073 %
12 6.3467 ± 0.1008 0.00318 ± 0.00086 0.00485 ± 0.00104 1.780 ± 0.217 % 98.775 ± 0.070 %
13 6.4863 ± 0.1007 0.00352 ± 0.00083 0.00486 ± 0.00097 1.836 ± 0.214 % 98.730 ± 0.069 %
14 6.6796 ± 0.1006 0.00357 ± 0.00079 0.00456 ± 0.00090 1.781 ± 0.205 % 98.730 ± 0.066 %
15 6.6054 ± 0.0956 0.00336 ± 0.00074 0.00433 ± 0.00084 1.731 ± 0.197 % 98.733 ± 0.064 %
16 6.4418 ± 0.0896 0.00324 ± 0.00071 0.00424 ± 0.00080 1.777 ± 0.196 % 98.730 ± 0.062 %
1.07.274.910 E ggml_cuda_compute_forward: MUL_MAT failed
1.07.275.095 E CUDA error: an illegal instruction was encountered
1.07.275.100 E current device: 0, in function ggml_cuda_compute_forward at ggml/src/ggml-cuda/ggml-cuda.cu:2547
1.07.275.100 E err
ggml/src/ggml-cuda/ggml-cuda.cu:113: CUDA error

Working log with -fit off

With -fit off the same command completes all 72 chunks and prints the KLD statistics block. The log ends with:

====== KL divergence statistics ======
Mean KLD: 0.008461 ± 0.000782
99.9% KLD: 1.111757
99.0% KLD: 0.012812
95.0% KLD: 0.003030
Median KLD: 0.000358

What I would like to see

llama-perplexity --kl-divergence completes successfully with -fit on (the default), so KLD benchmarks work out of the box.
Ideally also with KVarN cache types (-ctk kvarn4 -ctv kvarn4), which I haven't been able to test yet because the crash happens before finishing.

Lingua principale
C++
Stelle
1.2k
Fork
80
Merge medio
4g 20h
PR unite (30g)
2

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Anbeeld/beellama.cpp

Tutte le issue di Anbeeld/beellama.cpp

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.