/v1/models/capabilities -- wrong default for context_size
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 68/100
Research direction
Start at the /v1/models/capabilities endpoint and trace how context_size is selected when a model YAML has no value. Verify the behavior with the configured Settings → Performance → Default Context Size, and consider the issue done when the endpoint returns that default instead of 4096.
Written by the indexing model from the issue text.
Description
LocalAI version:
docker.io/localai/localai:latest-gpu-hipblas (v4.10.0 (7ad0cbf259f0c7bf9920fe2438fc3630ecd6c672))
Environment, CPU architecture, OS, and Version:
Linux, amd64, PikaOS (Debian Testing-based), latest
Describe the bug
The API endpoint /v1/models/capabilities returns 4096 for a model that has no context_size configured. I would expect Settings -> Performance -> Default Context Size to be used.
To Reproduce
- configure a model with no context_size set in its YAML config
- Go to
hostname/v1/models/capabilities
Expected behavior
Default context size is used when a model is not configured
Logs
not relevant
Additional context
🐈
- Dominant language
- Go
- Stars
- 49.2k
- Forks
- 4.5k
- Avg merge
- 19h 42m
- Merged PRs (30d)
- 299
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from mudler/LocalAI
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
mudler/LocalAI#11995 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
mudler/LocalAI#11991 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
mudler/LocalAI#11973 · 1 comment ·
Maintainers usually reply within 3 days
-
Quantization import always writes backend: llama-cpp, ignoring the backend that produced the modelOpen
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 3 days
Similar issues
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
volodya-lombrozo/aidy#333 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
rossoctl/context-guru#314 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 86/100
OwO-Network/DLX#236 · 1 comment · 1 reaction ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100