[BUG] llama-perplexity crashes with CUDA illegal instruction on RTX 3090 (--kl-divergence, -fit on)

Abierto
#187 1 comentario 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
cpp
Área
ai

Línea de trabajo

The crash is raised in ggml_cuda_compute_forward at ggml/src/ggml-cuda/ggml-cuda.cu line 2547, during a MUL_MAT op, and only with -fit on. Start by reproducing with llama-perplexity --kl-divergence and the -fit on versus -fit off flags, then compare which tensors and devices differ between the two runs. Done means the default -fit on run completes all chunks with KLD statistics printed.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Name and Version
  • Version: 0.4.8 (build 12480, commit ce1a9994f)
  • Built with: Clang 20.1.8 for Windows x86_64
  • ggml op count: 105
  • SHA256: 476353A6BB06D59578DC6E607466AC799AE74B03B1A9089A03B9912C0A185EF8
  • Build date: 2026-10-14 23:34:54
Operating systems

Windows

GGML backends

CUDA

Hardware

System:

  • OS: Windows 11 IoT Enterprise LTSC 10.0.26100
  • CPU: AMD Ryzen 9 5950X (16C/32T)
  • RAM: 64 GB
  • Shell: PowerShell 5.1.26100.9711

GPU:

  • Model: NVIDIA GeForce RTX 3090 (Ampere, compute capability 8.6)
  • VRAM: 24576 MiB
  • Driver: 617.42
  • CUDA UMD Version: 13.4
Models
Problem description & steps to reproduce

Summary

llama-perplexity.exe crashes with CUDA error: an illegal instruction was encountered when computing KL divergence (--kl-divergence + --kl-divergence-base) on Qwen3.8-27B IQ4_XS with -fit on (default). The same command with -fit off completes all 72 chunks successfully. Mainline llama.cpp (ggml-org) runs the same command to completion without errors in both modes.

Reproduction

Crashing variant (-fit on, the default — no flag needed):

.\llama-perplexity.exe `
    -m "<MODEL_DIR>\Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf" `
    -f "<MODEL_DIR>\wiki.test.raw" `
    -fa on `
    -ngl 99 `
    -c 4096 `
    --kl-divergence-base "<MODEL_DIR>\kld-baseline-fp16.kld" `
    --kl-divergence `
    --chunks 10 `
    --seed 1337 `
    --threads 1

Working variant (-fit off):

.\llama-perplexity.exe `
    -m "<MODEL_DIR>\Qwen3.8-27B-MTP-IMAT-IQ4_XS-Q8nextn.gguf" `
    -f "<MODEL_DIR>\wiki.test.raw" `
    -fa on `
    -ngl 99 `
    -c 4096 `
    -fit off `
    --kl-divergence-base "<MODEL_DIR>\kld-baseline-fp16.kld" `
    --kl-divergence `
    --chunks 10 `
    --seed 1337 `
    --threads 1

Baseline was created the same way without --kl-divergence-base/--kl-divergence.

Expected behavior

llama-perplexity --kl-divergence should complete successfully in both modes (-fit on and -fit off), as it does on ggml-org/llama.cpp.

Preliminary investigation (as requested in the template)

I ran the requested llama-completion -fit off test and both llama-perplexity variants.

Test-fit on``-fit off``llama-completion✅ PASS✅ PASSllama-perplexity --kl-divergence❌ CRASH (chunk 16)✅ PASS (all 72 chunks)
Conclusion: the crash is tied to -fit on in combination with --kl-divergence. Base inference (llama-completion) works fine in both modes.

Additional context

The same model, same --kl-divergence-base file, same arguments run to completion on ggml-org/llama.cpp (no crash, KLD statistics printed normally).
The bug is specific to BeeLlama 0.4.8 with -fit on.
Workaround: use -fit off when running llama-perplexity --kl-divergence.
I have not been able to test KVarN cache types (-ctk kvarn4 -ctv kvarn4) at all, because llama-perplexity crashes before finishing — which is the main reason I'm reporting this.

First Bad Commit

No response

Relevant log output

Crash log with -fit on (default)

Last 30 lines before crash:

10 6.1698 ± 0.1068 0.00337 ± 0.00101 0.00560 ± 0.00124 1.894 ± 0.244 % 98.764 ± 0.077 %
11 6.3279 ± 0.1055 0.00360 ± 0.00093 0.00523 ± 0.00113 1.848 ± 0.228 % 98.779 ± 0.073 %
12 6.3467 ± 0.1008 0.00318 ± 0.00086 0.00485 ± 0.00104 1.780 ± 0.217 % 98.775 ± 0.070 %
13 6.4863 ± 0.1007 0.00352 ± 0.00083 0.00486 ± 0.00097 1.836 ± 0.214 % 98.730 ± 0.069 %
14 6.6796 ± 0.1006 0.00357 ± 0.00079 0.00456 ± 0.00090 1.781 ± 0.205 % 98.730 ± 0.066 %
15 6.6054 ± 0.0956 0.00336 ± 0.00074 0.00433 ± 0.00084 1.731 ± 0.197 % 98.733 ± 0.064 %
16 6.4418 ± 0.0896 0.00324 ± 0.00071 0.00424 ± 0.00080 1.777 ± 0.196 % 98.730 ± 0.062 %
1.07.274.910 E ggml_cuda_compute_forward: MUL_MAT failed
1.07.275.095 E CUDA error: an illegal instruction was encountered
1.07.275.100 E current device: 0, in function ggml_cuda_compute_forward at ggml/src/ggml-cuda/ggml-cuda.cu:2547
1.07.275.100 E err
ggml/src/ggml-cuda/ggml-cuda.cu:113: CUDA error

Working log with -fit off

With -fit off the same command completes all 72 chunks and prints the KLD statistics block. The log ends with:

====== KL divergence statistics ======
Mean KLD: 0.008461 ± 0.000782
99.9% KLD: 1.111757
99.0% KLD: 0.012812
95.0% KLD: 0.003030
Median KLD: 0.000358

What I would like to see

llama-perplexity --kl-divergence completes successfully with -fit on (the default), so KLD benchmarks work out of the box.
Ideally also with KVarN cache types (-ctk kvarn4 -ctv kvarn4), which I haven't been able to test yet because the crash happens before finishing.

Lenguaje dominante
C++
Estrellas
1.2k
Forks
80
Merge medio
4 d 20 h
PR fusionados (30 d)
2

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Anbeeld/beellama.cpp

Todos los issues de Anbeeld/beellama.cpp

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.