[BUG] llama-perplexity crashes with CUDA illegal instruction on RTX 3090 (--kl-divergence, -fit on)

Ouverte
#187 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
25/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
cpp
Domaine
ai

Piste de recherche

The crash is raised in ggml_cuda_compute_forward at ggml/src/ggml-cuda/ggml-cuda.cu line 2547, during a MUL_MAT op, and only with -fit on. Start by reproducing with llama-perplexity --kl-divergence and the -fit on versus -fit off flags, then compare which tensors and devices differ between the two runs. Done means the default -fit on run completes all chunks with KLD statistics printed.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Name and Version
  • Version: 0.4.8 (build 12480, commit ce1a9994f)
  • Built with: Clang 20.1.8 for Windows x86_64
  • ggml op count: 105
  • SHA256: 476353A6BB06D59578DC6E607466AC799AE74B03B1A9089A03B9912C0A185EF8
  • Build date: 2026-10-14 23:34:54
Operating systems

Windows

GGML backends

CUDA

Hardware

System:

  • OS: Windows 11 IoT Enterprise LTSC 10.0.26100
  • CPU: AMD Ryzen 9 5950X (16C/32T)
  • RAM: 64 GB
  • Shell: PowerShell 5.1.26100.9711

GPU:

  • Model: NVIDIA GeForce RTX 3090 (Ampere, compute capability 8.6)
  • VRAM: 24576 MiB
  • Driver: 617.42
  • CUDA UMD Version: 13.4
Models
Problem description & steps to reproduce

Summary

llama-perplexity.exe crashes with CUDA error: an illegal instruction was encountered when computing KL divergence (--kl-divergence + --kl-divergence-base) on Qwen3.8-27B IQ4_XS with -fit on (default). The same command with -fit off completes all 72 chunks successfully. Mainline llama.cpp (ggml-org) runs the same command to completion without errors in both modes.

Reproduction

Crashing variant (-fit on, the default — no flag needed):

.\llama-perplexity.exe `
    -m "<MODEL_DIR>\Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf" `
    -f "<MODEL_DIR>\wiki.test.raw" `
    -fa on `
    -ngl 99 `
    -c 4096 `
    --kl-divergence-base "<MODEL_DIR>\kld-baseline-fp16.kld" `
    --kl-divergence `
    --chunks 10 `
    --seed 1337 `
    --threads 1

Working variant (-fit off):

.\llama-perplexity.exe `
    -m "<MODEL_DIR>\Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf" `
    -f "<MODEL_DIR>\wiki.test.raw" `
    -fa on `
    -ngl 99 `
    -c 4096 `
    -fit off `
    --kl-divergence-base "<MODEL_DIR>\kld-baseline-fp16.kld" `
    --kl-divergence `
    --chunks 10 `
    --seed 1337 `
    --threads 1

Baseline was created the same way without --kl-divergence-base/--kl-divergence.

Expected behavior

llama-perplexity --kl-divergence should complete successfully in both modes (-fit on and -fit off), as it does on ggml-org/llama.cpp.

Preliminary investigation (as requested in the template)

I ran the requested llama-completion -fit off test and both llama-perplexity variants.

Test-fit on``-fit off``llama-completion✅ PASS✅ PASSllama-perplexity --kl-divergence❌ CRASH (chunk 16)✅ PASS (all 72 chunks)
Conclusion: the crash is tied to -fit on in combination with --kl-divergence. Base inference (llama-completion) works fine in both modes.

Additional context

The same model, same --kl-divergence-base file, same arguments run to completion on ggml-org/llama.cpp (no crash, KLD statistics printed normally).
The bug is specific to BeeLlama 0.4.8 with -fit on.
Workaround: use -fit off when running llama-perplexity --kl-divergence.
I have not been able to test KVarN cache types (-ctk kvarn4 -ctv kvarn4) at all, because llama-perplexity crashes before finishing — which is the main reason I'm reporting this.

First Bad Commit

No response

Relevant log output

Crash log with -fit on (default)

Last 30 lines before crash:

10 6.1698 ± 0.1068 0.00337 ± 0.00101 0.00560 ± 0.00124 1.894 ± 0.244 % 98.764 ± 0.077 %
11 6.3279 ± 0.1055 0.00360 ± 0.00093 0.00523 ± 0.00113 1.848 ± 0.228 % 98.779 ± 0.073 %
12 6.3467 ± 0.1008 0.00318 ± 0.00086 0.00485 ± 0.00104 1.780 ± 0.217 % 98.775 ± 0.070 %
13 6.4863 ± 0.1007 0.00352 ± 0.00083 0.00486 ± 0.00097 1.836 ± 0.214 % 98.730 ± 0.069 %
14 6.6796 ± 0.1006 0.00357 ± 0.00079 0.00456 ± 0.00090 1.781 ± 0.205 % 98.730 ± 0.066 %
15 6.6054 ± 0.0956 0.00336 ± 0.00074 0.00433 ± 0.00084 1.731 ± 0.197 % 98.733 ± 0.064 %
16 6.4418 ± 0.0896 0.00324 ± 0.00071 0.00424 ± 0.00080 1.777 ± 0.196 % 98.730 ± 0.062 %
1.07.274.910 E ggml_cuda_compute_forward: MUL_MAT failed
1.07.275.095 E CUDA error: an illegal instruction was encountered
1.07.275.100 E current device: 0, in function ggml_cuda_compute_forward at ggml/src/ggml-cuda/ggml-cuda.cu:2547
1.07.275.100 E err
ggml/src/ggml-cuda/ggml-cuda.cu:113: CUDA error

Working log with -fit off

With -fit off the same command completes all 72 chunks and prints the KLD statistics block. The log ends with:

====== KL divergence statistics ======
Mean KLD: 0.008461 ± 0.000782
99.9% KLD: 1.111757
99.0% KLD: 0.012812
95.0% KLD: 0.003030
Median KLD: 0.000358

What I would like to see

llama-perplexity --kl-divergence completes successfully with -fit on (the default), so KLD benchmarks work out of the box.
Ideally also with KVarN cache types (-ctk kvarn4 -ctv kvarn4), which I haven't been able to test yet because the crash happens before finishing.

Langage dominant
C++
Étoiles
1.2k
Forks
80
Merge moyen
4 j 20 h
PR mergées (30 j)
2

Préparer son environnement

Ce projet ne fournit ni conteneur de développement, ni Dockerfile, ni guide de contribution : l'installation est à votre charge. Commencez par son README, et consultez notre guide de la première contribution pour les étapes générales.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de Anbeeld/beellama.cpp

Toutes les issues de Anbeeld/beellama.cpp

Issues similaires

Plus d'issues C++

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.