Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

cuda13-llama-cpp backend crashes loading gemma4 GGUF: "wrong number of tensors; expected 2131, got 720"

Abierto
  • ¿Algo no está bien?
#11,593 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Tranquilo
Área
backend

Línea de trabajo

Comienza con la ruta de carga del backend de cuda13-llama-cpp y reproduce el fallo usando el gemma4 GGUF proporcionado y la solicitud POST /v1/chat/completions. Comprueba cómo el loader gestiona la discrepancia en el número de tensores y la salida del backend; se considera terminado si se logra cargar correctamente o se obtiene un error limpio de arquitectura no compatible sin que muera el proceso.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Description

Loading a gemma4 (Google Gemma 4, e.g. gemma4:e4b) architecture GGUF via the
cuda13-llama-cpp backend fails to load, with the backend process crashing
(exitCode -1) after emitting a tensor-count mismatch from llama.cpp's loader.

This blocks migrating any Ollama-served gemma4-family model onto LocalAI's
llama-cpp backend, since llama.cpp itself is where the incompatibility lives (see
"Root cause" below) — LocalAI is just the messenger here, but the crash-on-load
behavior (full backend process death instead of a clean load-time model
validation error) is a LocalAI/backend robustness gap worth tracking regardless
of when upstream llama.cpp lands full gemma4 support.

LocalAI version

v4.8.2 (5ff25d9d145e0a03a5b9a3559c620f1e1204ca6d), running via Docker image
localai/localai:latest-gpu-nvidia-cuda-13.

Backend: cuda13-llama-cpp, installed today via the gallery from
quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-llama-cpp
(digest sha256:389b92f825136696f85c548b9dcda65c87a224261d15f0089772dd1d40d5fda0).

Environment
  • Docker container localai/localai:latest-gpu-nvidia-cuda-13, --gpus all,
    NVIDIA RTX 5080, nvidia-container-toolkit GPU passthrough verified working
    (other models — a llama-3.2-1b test model and a Qwen3.6-35B-A3B MoE model —
    load and infer correctly on this same instance via the same cuda13-llama-cpp
    backend).
  • Host: Linux x86_64.
Model / GGUF
  • general.architecture (read directly from GGUF header): gemma4
  • GGUF header: tensor_count = 2131, kv_count = 55, GGUF version 3
  • This is the exact GGUF blob Ollama 0.32.6 ships/uses for its gemma4:e4b tag
    (extracted byte-identical, sha256-verified, directly from Ollama's own blob
    store — same file Ollama's own bundled llama-server process currently loads
    and serves successfully with --no-jinja --chat-template chatml).
  • Quant: Q4_K_M, ~9.6GB file size, ~8.19B params.
Steps to reproduce
  1. Place the GGUF under LocalAI's models directory.
  2. Model YAML:
name: gemma4-e4b
backend: cuda13-llama-cpp
f16: true
mmap: true
context_size: 4096
parameters:
  model: gemma4-e4b-q4_k_m.gguf
options:
  - jinja
  1. POST /v1/chat/completions with "model": "gemma4-e4b".
Expected behavior

Either the model loads and serves a completion, or LocalAI returns a clean
"unsupported architecture" error without killing the backend process.

Actual behavior

Backend process crashes immediately on load attempt (exitCode -1), and the API
returns a 500 with this llama.cpp loader error:

Failed to load model: /models/gemma4-e4b-q4_k_m.gguf. Error: llama_model_load:
error loading model: done_getting_tensors: wrong number of tensors; expected
2131, got 720; llama_model_load_from_file_impl: failed to load model;
llama_model_load: error loading model: done_getting_tensors: wrong number of
tensors; expected 2131, got 720; llama_model_load_from_file_impl: failed to
load model

Full log excerpt:

INFO  BackendLoader starting modelID=gemma4-e4b backend=cuda13-llama-cpp model=gemma4-e4b-q4_k_m.gguf
INFO  effective runtime tuning ... context=4096 n_batch=2048 n_gpu_layers=99999999 parallel=1 flash_attention=auto f16=true
WARN  Backend process exited unexpectedly id=gemma4-e4b address=127.0.0.1:34875 process=run.sh exitCode=-1
ERROR Failed to load model modelID=gemma4-e4b error=failed to load model with internal loader: could not load model: rpc error: code = Internal desc = Failed to load model: ... (as above)
Root cause (as best I can tell)

The bundled llama.cpp in this backend build only partially recognizes the
gemma4 GGUF architecture — it maps 720 of the declared 2131 tensors before
giving up, rather than either fully supporting the architecture or cleanly
rejecting it up front. This tracks with upstream ggml-org/llama.cpp gemma4
support still being in active flux — several open issues/PRs there reference
gemma4-family loading/eval problems, e.g.:

None of those match this exact "wrong number of tensors; expected 2131, got
720" symptom, so this looks like a distinct manifestation of the same
underlying "gemma4 architecture support isn't finished in llama.cpp yet" gap,
surfaced through LocalAI's bundled backend.

Ask
  1. If this is purely an upstream llama.cpp support gap, it'd help to have that
    stated/tracked here (e.g. a pointer to which llama.cpp PR/milestone brings
    full gemma4 tensor support) so downstream LocalAI users know when to
    retry.
  2. Separately: LocalAI/the backend should not let this crash the whole backend
    process — a clean "unsupported architecture" load-time error (instead of
    exitCode -1 process death class of failure) would make this kind of gap
    much easier to diagnose from the API side alone, without digging through
    container logs.

Happy to provide the full GGUF metadata dump or test further builds if useful.

Lenguaje dominante
Go
Estrellas
49.2k
Forks
4.5k
Merge medio
1 d 7 h
PR fusionados (30 d)
340

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de mudler/LocalAI

Todos los issues de mudler/LocalAI

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.