bug: StaticServiceDiscovery validates one of four index-aligned lists, so a mismatch gives an IndexError at request time and a truncated health-check pass
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 84/100
Research direction
Start in src/vllm_router/service_discovery.py at StaticServiceDiscovery and inspect its constructor, get_endpoint_info(), and get_unhealthy_endpoint_hashes(). Reproduce the provided mismatched-list example first. Done means every index-aligned list mismatch fails during construction with list names and lengths, while an incomplete health check returns no partial endpoint list.
Written by the indexing model from the issue text.
Description
Describe the bug
StaticServiceDiscovery takes four index-aligned lists and validates one pair of them. The constructor accepts a mismatch in
either of the other two, and it surfaces later, in two ways, neither of them a clear configuration error.
service_discovery.py:237 asserts the first pair only:
assert len(urls) == len(models), "URLs and models should have the same length"
It stores model_labels and model_types unchecked. Then:
get_endpoint_info()indexesself.model_labels[i], so a short--static-model-labelsraises
IndexErroron a request rather than at startup.get_unhealthy_endpoint_hashes()iterates
zip(self.urls, self.models, self.model_types, strict=True)inside atrythat catchesValueError.
The loop appends as it runs, so the exception arrives after the endpoints ahead of the mismatch
already sit in the list, and the caller receives that partial list as though it were complete. The
log says health checks were skipped.
aliases stays out of this. :372 uses it as a membership test rather than by index, so its length has
no relation to len(urls).
To Reproduce
import sys; sys.path.insert(0, "src")
from vllm_router.service_discovery import StaticServiceDiscovery
sd = StaticServiceDiscovery(
app=None,
urls=["http://127.0.0.1:1", "http://127.0.0.1:2"],
models=["m-a", "m-b"],
model_labels=["only-one"],
model_types=["chat"],
)
print("constructed, no error")
try:
sd.get_endpoint_info()
except IndexError as exc:
print("get_endpoint_info raised IndexError:", exc)
print("get_unhealthy_endpoint_hashes ->", sd.get_unhealthy_endpoint_hashes())
Expected behavior
A length mismatch across the index-aligned lists fails at construction, with a message naming the two
lists and their lengths. A health-check pass that cannot complete returns nothing rather than a partial
result, so the router does not treat an unchecked backend as healthy.
Additional context
Measured on 6a33ae4.
constructed, no error
get_endpoint_info raised IndexError: list index out of range
model_label = self.model_labels[i] if self.model_labels else "default" # service_discovery.py:333
WARNING m-a at http://127.0.0.1:1 not healthy!
ERROR To perform health check, each model has to define a static_model_type and at least one
static_backend. Skipping health checks for now.
get_unhealthy_endpoint_hashes -> ['ee047b9682fb943f3f82e0b83ffa2635']
The router drops m-a, correctly, since nothing listens on that port. It never checks m-b, so m-b
stays in rotation on the strength of a health check that did not run.
Provenance, so nobody reads this as a second venue for one discussion. Both halves appear in my own two
comments on #1049
(first,
correction). #1049
is sethforprivacy's feature request for advertising max_model_len, and its proposed
--static-model-lens would be the fifth index-aligned list. I filed it separately so that fixing the validation does not
wait on that feature, and so the feature request keeps its own subject.
I did not check the Kubernetes or service-name discovery classes, which build their model info from the
engine rather than from index-aligned lists.
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 503
- Avg merge
- 4d 17h
- Merged PRs (30d)
- 8
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from vllm-project/production-stack
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
vllm-project/production-stack#1057 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
vllm-project/production-stack#981 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 70/100
vllm-project/production-stack#1091 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
vllm-project/production-stack#1087 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
vllm-project/production-stack#1074 ·
All issues in vllm-project/production-stack
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100