[FEATURE] A way to force swap models that are in a ModelPool and ModelRouter
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 22/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
Research direction
The issue asks for a way to force-swap a model in a ModelPool behind a ModelRouter, and the payload names no specific files or tests. Start by finding where the router selects and scales models in the pool, and reproduce the wrong-model swap the reporter describes. Done would mean a maintainer-agreed design (the long-term fix is unspecified) plus a manual override path that scales down the other pool members.
Written by the indexing model from the issue text.
Description
Feature Description
I would like a way to force swap a model that's part of a ModelPool.
Problem Statement
What problem does this feature solve? What use case does it enable?
Sometimes when there is a request going through the ModelRouter, I notice that the wrong model will get swapped in instead of the actual target model, and only after that the target model will get scaled up. Then, since both are scaled up but the wrong one got scaled up first, it insists on occupying the hardware requests instead of the correct target model.
Proposed Solution
The long term solution is to fix the issue with the swapping.
The short term solution that applies to other cases (such as experimentation or troubleshooting), is a way (CLI or kubectl patch something) to force swap a model, and scale down all other models that the forced model shares in its particpating ModelPool.
Priority
How important is this feature to you?
- Critical - Blocking my use case
- High - Would significantly improve my workflow
- Medium - Nice to have
- Low - Minor improvement
Willingness to Contribute
Are you willing to contribute to this feature?
- Yes, I can submit a PR
- Yes, I can help test
- No, but I can provide feedback
- No, just requesting
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day