[BUG] llama-perplexity crashes with CUDA illegal instruction on RTX 3090 (--kl-divergence, -fit on)
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 25/100
Línea de trabajo
The crash is raised in ggml_cuda_compute_forward at ggml/src/ggml-cuda/ggml-cuda.cu line 2547, during a MUL_MAT op, and only with -fit on. Start by reproducing with llama-perplexity --kl-divergence and the -fit on versus -fit off flags, then compare which tensors and devices differ between the two runs. Done means the default -fit on run completes all chunks with KLD statistics printed.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Name and Version
- Version: 0.4.8 (build 12480, commit ce1a9994f)
- Built with: Clang 20.1.8 for Windows x86_64
- ggml op count: 105
- SHA256: 476353A6BB06D59578DC6E607466AC799AE74B03B1A9089A03B9912C0A185EF8
- Build date: 2026-10-14 23:34:54
Operating systems
Windows
GGML backends
CUDA
Hardware
System:
- OS: Windows 11 IoT Enterprise LTSC 10.0.26100
- CPU: AMD Ryzen 9 5950X (16C/32T)
- RAM: 64 GB
- Shell: PowerShell 5.1.26100.9711
GPU:
- Model: NVIDIA GeForce RTX 3090 (Ampere, compute capability 8.6)
- VRAM: 24576 MiB
- Driver: 617.42
- CUDA UMD Version: 13.4
Models
- Name: Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf
- Source: https://huggingface.co/localweights/Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn-GGUF/resolve/main/Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf
- Quant: IQ4_XS (imatrix), with embedded NextN/MTP head (Q8_0)
- Architecture: Qwen 3.8 Dense 27B
- Note: this GGUF contains an embedded NextN/MTP head (Q8_0). llama-perplexity does not use MTP, but mentioning it in case the loader treats MTP-embedded files differently.
Problem description & steps to reproduce
Summary
llama-perplexity.exe crashes with CUDA error: an illegal instruction was encountered when computing KL divergence (--kl-divergence + --kl-divergence-base) on Qwen3.8-27B IQ4_XS with -fit on (default). The same command with -fit off completes all 72 chunks successfully. Mainline llama.cpp (ggml-org) runs the same command to completion without errors in both modes.
Reproduction
Crashing variant (-fit on, the default — no flag needed):
.\llama-perplexity.exe `
-m "<MODEL_DIR>\Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf" `
-f "<MODEL_DIR>\wiki.test.raw" `
-fa on `
-ngl 99 `
-c 4096 `
--kl-divergence-base "<MODEL_DIR>\kld-baseline-fp16.kld" `
--kl-divergence `
--chunks 10 `
--seed 1337 `
--threads 1
Working variant (-fit off):
.\llama-perplexity.exe `
-m "<MODEL_DIR>\Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf" `
-f "<MODEL_DIR>\wiki.test.raw" `
-fa on `
-ngl 99 `
-c 4096 `
-fit off `
--kl-divergence-base "<MODEL_DIR>\kld-baseline-fp16.kld" `
--kl-divergence `
--chunks 10 `
--seed 1337 `
--threads 1
Baseline was created the same way without --kl-divergence-base/--kl-divergence.
Expected behavior
llama-perplexity --kl-divergence should complete successfully in both modes (-fit on and -fit off), as it does on ggml-org/llama.cpp.
Preliminary investigation (as requested in the template)
I ran the requested llama-completion -fit off test and both llama-perplexity variants.
Test-fit on``-fit off``llama-completion✅ PASS✅ PASSllama-perplexity --kl-divergence❌ CRASH (chunk 16)✅ PASS (all 72 chunks)
Conclusion: the crash is tied to -fit on in combination with --kl-divergence. Base inference (llama-completion) works fine in both modes.
Additional context
The same model, same --kl-divergence-base file, same arguments run to completion on ggml-org/llama.cpp (no crash, KLD statistics printed normally).
The bug is specific to BeeLlama 0.4.8 with -fit on.
Workaround: use -fit off when running llama-perplexity --kl-divergence.
I have not been able to test KVarN cache types (-ctk kvarn4 -ctv kvarn4) at all, because llama-perplexity crashes before finishing — which is the main reason I'm reporting this.
First Bad Commit
No response
Relevant log output
Crash log with -fit on (default)
Last 30 lines before crash:
10 6.1698 ± 0.1068 0.00337 ± 0.00101 0.00560 ± 0.00124 1.894 ± 0.244 % 98.764 ± 0.077 %
11 6.3279 ± 0.1055 0.00360 ± 0.00093 0.00523 ± 0.00113 1.848 ± 0.228 % 98.779 ± 0.073 %
12 6.3467 ± 0.1008 0.00318 ± 0.00086 0.00485 ± 0.00104 1.780 ± 0.217 % 98.775 ± 0.070 %
13 6.4863 ± 0.1007 0.00352 ± 0.00083 0.00486 ± 0.00097 1.836 ± 0.214 % 98.730 ± 0.069 %
14 6.6796 ± 0.1006 0.00357 ± 0.00079 0.00456 ± 0.00090 1.781 ± 0.205 % 98.730 ± 0.066 %
15 6.6054 ± 0.0956 0.00336 ± 0.00074 0.00433 ± 0.00084 1.731 ± 0.197 % 98.733 ± 0.064 %
16 6.4418 ± 0.0896 0.00324 ± 0.00071 0.00424 ± 0.00080 1.777 ± 0.196 % 98.730 ± 0.062 %
1.07.274.910 E ggml_cuda_compute_forward: MUL_MAT failed
1.07.275.095 E CUDA error: an illegal instruction was encountered
1.07.275.100 E current device: 0, in function ggml_cuda_compute_forward at ggml/src/ggml-cuda/ggml-cuda.cu:2547
1.07.275.100 E err
ggml/src/ggml-cuda/ggml-cuda.cu:113: CUDA error
Working log with -fit off
With -fit off the same command completes all 72 chunks and prints the KLD statistics block. The log ends with:
====== KL divergence statistics ======
Mean KLD: 0.008461 ± 0.000782
99.9% KLD: 1.111757
99.0% KLD: 0.012812
95.0% KLD: 0.003030
Median KLD: 0.000358
What I would like to see
llama-perplexity --kl-divergence completes successfully with -fit on (the default), so KLD benchmarks work out of the box.
Ideally also with KVarN cache types (-ctk kvarn4 -ctv kvarn4), which I haven't been able to test yet because the crash happens before finishing.
- Lenguaje dominante
- C++
- Estrellas
- 1.2k
- Forks
- 80
- Merge medio
- 4 d 20 h
- PR fusionados (30 d)
- 2
Preparar el entorno
Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de Anbeeld/beellama.cpp
-
[BUG] KVarN assertion failure in build_inp_mem_hybrid_iswa() due to uninitialized self_kvarn_rotAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
Anbeeld/beellama.cpp#205 ·
-
Dificultad 3/5 Medio día Aptitud para principiantes 58/100
Anbeeld/beellama.cpp#206 ·
-
Dificultad 3/5 1-2 días Aptitud para principiantes 57/100
Anbeeld/beellama.cpp#204 ·
-
Dificultad 4/5 3-5 días Aptitud para principiantes 45/100
Anbeeld/beellama.cpp#203 · 2 comentarios ·
-
Dificultad 4/5 3-5 días Aptitud para principiantes 45/100
Anbeeld/beellama.cpp#202 · 1 comentario ·
Todos los issues de Anbeeld/beellama.cpp
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
CachyOS/packageinstaller#144 ·
Los mantenedores suelen responder en 1 día
-
documentation
Dificultad 1/5 Menos de una hora Aptitud para principiantes 82/100
TomerAberbach/profiler-md#951 ·
Los mantenedores suelen responder en 1 día
-
[Bug] Default key routing (JavaStringHash) picks a different partition than the Java client for about half of keysPosiblemente ocupada @0z5a la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
apache/pulsar-client-go#1575 ·
Los mantenedores suelen responder en 2 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
pocoproject/poco#5764 ·
Los mantenedores suelen responder en 2 días
-
bug cpp-core-guidelines low-priority
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
NatronGitHub/openfx-supportext#164 ·
Los mantenedores suelen responder en 1 día