Quantization import always writes backend: llama-cpp, ignoring the backend that produced the model
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
Research direction
Start at the POST /api/quantization/jobs/:id/import handler and inspect schema.QuantizationJob.Backend, then trace where the generated YAML sets backend and description. Reproduce an import with the rocmfp4 backend and verify the output preserves the job backend and no longer describes the model as fine-tuned.
Written by the indexing model from the issue text.
Description
Summary
POST /api/quantization/jobs/:id/import always writes backend: llama-cpp into the generated model config, regardless of the quantization type the same service just produced. For any type stock llama.cpp cannot read, the imported model is not loadable.
Reproduction
Quantize through /app/quantize with a backend whose format stock llama.cpp does not know, then import the result. The generated YAML:
parameters:
model: <name>/model-Q4_0_ROCMFP4_STRIX_LEAN.gguf
name: <name>
backend: llama-cpp # <- always this
template:
use_tokenizer_template: true
known_usecases:
- chat
description: Fine-tuned model (GGUF)
options:
- use_jinja:true
Loading it fails, because the tensor types are unknown to that backend. Correcting backend: by hand makes the same file work immediately.
Concretely this happened with the rocmfp4 backend (ROCmFP4 weight types, #11636), but the shape of the problem is general: the service knows which backend served StartQuantization, and discards that knowledge at import.
Suggestion
Carry the job's Backend into the imported config instead of defaulting. The value is already on the job record (schema.QuantizationJob.Backend), so the import has it in hand.
A second, smaller point in the same place: the description is hardcoded to Fine-tuned model (GGUF) for a model that was quantized, not fine-tuned.
Impact
Small in code, awkward in practice: the one path that produces a model and the one path that registers it disagree about how to run it, and the user finds out only when the model refuses to load.
Disclosure: this report was prepared with AI assistance (Claude); the import was run on a real host (LocalAI v4.9.0-305-g8351db5de, ROCm 7.x / gfx1151) and the YAML above is the file it wrote.
- Dominant language
- Go
- Stars
- 49.2k
- Forks
- 4.5k
- Avg merge
- 1d 3m
- Merged PRs (30d)
- 239
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from mudler/LocalAI
-
area/container bug confirmed kind/documentation
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
fish-speech: make compile:true usable on Blackwell sm_121 by honouring the CUDA toolkit's ptxas Openenhancement
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 84/100
-
enhancement needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
kind/cleanup
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
kubernetes-sigs/kueue#15947 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
sympozium-ai/sympozium#627 ·
-
priority: p3
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
googleapis/librarian#7636 ·