Feature: opt-in failover for embeddings virtual models
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
Research direction
Read tryFailoverEmbeddings in internal/gateway/inference_execute.go and trace how virtual model configuration and failover targets are handled. The issue proposes an opt-in such as embeddings_failover, with compatibility asserted by the operator; check existing virtual model config and gateway tests to find the relevant entry points. Done means opted-in embedding requests can fail over while the default still prevents incompatible vectors.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem? Please describe.
Failover is skipped for /v1/embeddings (tryFailoverEmbeddings in internal/gateway/inference_execute.go returns the primary error: "intentionally disabled until the shared model contract can prove vector-size compatibility"). I understand the concern, but it makes a strategy: failover virtual model useless for embeddings when the operator knows the targets serve the same model. Our setup: two self-hosted OpenAI-compatible servers (oMLX on two Macs) both serving EmbeddingGemma-2; when the primary is off, embedding requests fail instead of going to the second server.
Describe the solution you'd like
An explicit opt-in on the virtual model, e.g.:
virtual_models:
- source: embed
strategy: failover
embeddings_failover: true # operator asserts targets produce compatible vectors
targets:
- {model: server1/EmbeddingGemma-2}
- {model: server2/EmbeddingGemma-2}
Optionally, GoModel could still guard it automatically: allow failover when all targets share the same upstream model id, or compare the returned dimension with the primary's last known dimension and refuse a mismatch.
Describe alternatives you've considered
Declaring no model list per provider so a down server disappears from the inventory and the alias resolves to the live one. It works, but only after the inventory refresh (up to refresh_interval), and not for a server that fails while still listed.
Additional context
Migration from LiteLLM, where order-based fallback applies to embeddings as well. This was one of the two blockers for a like-for-like replacement.
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 111
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 123
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ENTERPILOT/GoModel
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 15/100
ENTERPILOT/GoModel#1290 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
ENTERPILOT/GoModel#1287 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 46/100
ENTERPILOT/GoModel#1285 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 28/100
ENTERPILOT/GoModel#1249 ·
Maintainers usually reply within 1 day
-
Merge usage analytics from multiple GoModel instances with separate databasesPossibly taken @SantiagoDePolonia claimed this 3 days ago. Open
Difficulty 5/5 Over a week Newbie friendliness 35/100
ENTERPILOT/GoModel#1247 ·
Maintainers usually reply within 1 day
All issues in ENTERPILOT/GoModel
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
ovh/ovhcloud-cli#309 ·
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
area/docs
Difficulty 1/5 Under an hour Newbie friendliness 92/100
layer5io/getnighthawk#380 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day