[FEATURE] A way to force swap models that are in a ModelPool and ModelRouter

Open
#2,533 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
22/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes

Research direction

The issue asks for a way to force-swap a model in a ModelPool behind a ModelRouter, and the payload names no specific files or tests. Start by finding where the router selects and scales models in the pool, and reproduce the wrong-model swap the reporter describes. Done would mean a maintainer-agreed design (the long-term fix is unspecified) plus a manual override path that scales down the other pool members.

Written by the indexing model from the issue text.

Description

area/routing component/controller enhancement kind/feature priority/medium size/small

Feature Description

I would like a way to force swap a model that's part of a ModelPool.

Problem Statement

What problem does this feature solve? What use case does it enable?

Sometimes when there is a request going through the ModelRouter, I notice that the wrong model will get swapped in instead of the actual target model, and only after that the target model will get scaled up. Then, since both are scaled up but the wrong one got scaled up first, it insists on occupying the hardware requests instead of the correct target model.

Proposed Solution

The long term solution is to fix the issue with the swapping.

The short term solution that applies to other cases (such as experimentation or troubleshooting), is a way (CLI or kubectl patch something) to force swap a model, and scale down all other models that the forced model shares in its particpating ModelPool.

Priority

How important is this feature to you?

  • Critical - Blocking my use case
  • High - Would significantly improve my workflow
  • Medium - Nice to have
  • Low - Minor improvement

Willingness to Contribute

Are you willing to contribute to this feature?

  • Yes, I can submit a PR
  • Yes, I can help test
  • No, but I can provide feedback
  • No, just requesting
Dominant language
Go
Stars
232
Forks
37
Avg merge
21h 42m
Merged PRs (30d)
244

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from defilantech/LLMKube

All issues in defilantech/LLMKube

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.