[FEATURE] Read the GGUF pooling_type so embedding mode can fall back when it is missing

Open
#2,580 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes
Domain
ai, backend

Research direction

Start by reading the GGUF metadata parser in pkg/gguf and the controller and Metal agent llama.cpp argument builders; the issue names these entry points but not their file paths. Trace how Model metadata and extraArgs reach both builders, then inspect the existing parity test. Done means pooling metadata is read and the proposed default behavior is covered consistently in both builders, including the case where the user supplies --pooling.

Written by the indexing model from the issue text.

Description

enhancement

Feature Description

Read the GGUF's <arch>.pooling_type into the Model status, so an embedding service whose GGUF lacks it can get a working default instead of failing.

Problem Statement

With #2549 (fixing #2539), embedding mode no longer forces --pooling last: llama-server takes the pooling type from the GGUF's <arch>.pooling_type. When a GGUF doesn't carry that key, llama-server's default is pooling none, and /v1/embeddings then rejects every request ("Pooling type 'none' is not OAI compatible") while /health still passes. Today the only fix is extraArgs: ["--pooling", "last"], which a user finds only after their embeddings break.

Proposed Solution

  • Parse <arch>.pooling_type in pkg/gguf into GGUFMetadata, beside context_length and block_count.
  • In both llama.cpp argument builders (the controller's and the metal agent's), add a default --pooling only when the Model's metadata has no pooling type and the user set none in extraArgs. last was the old default; whether it should depend on the architecture is part of the design.
  • Keep the parity test between the two builders covering the new case.

Alternatives Considered

  • Leave it to extraArgs and the docs, as #2549 does.
  • A pooling field on the InferenceService spec.

Additional Context

Follow-up to the review of #2549.

Priority

  • Critical
  • High
  • Medium
  • Low

Willingness to Contribute

  • I can help with implementation
Dominant language
Go
Stars
232
Forks
37
Avg merge
21h 42m
Merged PRs (30d)
244

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from defilantech/LLMKube

All issues in defilantech/LLMKube

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.