[FEATURE] Read the GGUF pooling_type so embedding mode can fall back when it is missing
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
Research direction
Start by reading the GGUF metadata parser in pkg/gguf and the controller and Metal agent llama.cpp argument builders; the issue names these entry points but not their file paths. Trace how Model metadata and extraArgs reach both builders, then inspect the existing parity test. Done means pooling metadata is read and the proposed default behavior is covered consistently in both builders, including the case where the user supplies --pooling.
Written by the indexing model from the issue text.
Description
Feature Description
Read the GGUF's <arch>.pooling_type into the Model status, so an embedding service whose GGUF lacks it can get a working default instead of failing.
Problem Statement
With #2549 (fixing #2539), embedding mode no longer forces --pooling last: llama-server takes the pooling type from the GGUF's <arch>.pooling_type. When a GGUF doesn't carry that key, llama-server's default is pooling none, and /v1/embeddings then rejects every request ("Pooling type 'none' is not OAI compatible") while /health still passes. Today the only fix is extraArgs: ["--pooling", "last"], which a user finds only after their embeddings break.
Proposed Solution
- Parse
<arch>.pooling_typeinpkg/ggufintoGGUFMetadata, besidecontext_lengthandblock_count. - In both llama.cpp argument builders (the controller's and the metal agent's), add a default
--poolingonly when the Model's metadata has no pooling type and the user set none inextraArgs.lastwas the old default; whether it should depend on the architecture is part of the design. - Keep the parity test between the two builders covering the new case.
Alternatives Considered
- Leave it to
extraArgsand the docs, as #2549 does. - A
poolingfield on the InferenceService spec.
Additional Context
Follow-up to the review of #2549.
Priority
- Critical
- High
- Medium
- Low
Willingness to Contribute
- I can help with implementation
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day