bug: StaticServiceDiscovery validates one of four index-aligned lists, so a mismatch gives an IndexError at request time and a truncated health-check pass

Open Beginner friendly
#1,083 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend

Research direction

Start in src/vllm_router/service_discovery.py at StaticServiceDiscovery and inspect its constructor, get_endpoint_info(), and get_unhealthy_endpoint_hashes(). Reproduce the provided mismatched-list example first. Done means every index-aligned list mismatch fails during construction with list names and lengths, while an incomplete health check returns no partial endpoint list.

Written by the indexing model from the issue text.

Description

Describe the bug

StaticServiceDiscovery takes four index-aligned lists and validates one pair of them. The constructor accepts a mismatch in
either of the other two, and it surfaces later, in two ways, neither of them a clear configuration error.

service_discovery.py:237 asserts the first pair only:

assert len(urls) == len(models), "URLs and models should have the same length"

It stores model_labels and model_types unchecked. Then:

  1. get_endpoint_info() indexes self.model_labels[i], so a short --static-model-labels raises
    IndexError on a request rather than at startup.
  2. get_unhealthy_endpoint_hashes() iterates
    zip(self.urls, self.models, self.model_types, strict=True) inside a try that catches ValueError.
    The loop appends as it runs, so the exception arrives after the endpoints ahead of the mismatch
    already sit in the list, and the caller receives that partial list as though it were complete. The
    log says health checks were skipped.

aliases stays out of this. :372 uses it as a membership test rather than by index, so its length has
no relation to len(urls).

To Reproduce
import sys; sys.path.insert(0, "src")
from vllm_router.service_discovery import StaticServiceDiscovery

sd = StaticServiceDiscovery(
    app=None,
    urls=["http://127.0.0.1:1", "http://127.0.0.1:2"],
    models=["m-a", "m-b"],
    model_labels=["only-one"],
    model_types=["chat"],
)
print("constructed, no error")

try:
    sd.get_endpoint_info()
except IndexError as exc:
    print("get_endpoint_info raised IndexError:", exc)

print("get_unhealthy_endpoint_hashes ->", sd.get_unhealthy_endpoint_hashes())
Expected behavior

A length mismatch across the index-aligned lists fails at construction, with a message naming the two
lists and their lengths. A health-check pass that cannot complete returns nothing rather than a partial
result, so the router does not treat an unchecked backend as healthy.

Additional context

Measured on 6a33ae4.

constructed, no error
get_endpoint_info raised IndexError: list index out of range
    model_label = self.model_labels[i] if self.model_labels else "default"   # service_discovery.py:333
WARNING m-a at http://127.0.0.1:1 not healthy!
ERROR   To perform health check, each model has to define a static_model_type and at least one
        static_backend. Skipping health checks for now.
get_unhealthy_endpoint_hashes -> ['ee047b9682fb943f3f82e0b83ffa2635']

The router drops m-a, correctly, since nothing listens on that port. It never checks m-b, so m-b
stays in rotation on the strength of a health check that did not run.

Provenance, so nobody reads this as a second venue for one discussion. Both halves appear in my own two
comments on #1049
(first,
correction). #1049
is sethforprivacy's feature request for advertising max_model_len, and its proposed
--static-model-lens would be the fifth index-aligned list. I filed it separately so that fixing the validation does not
wait on that feature, and so the feature request keeps its own subject.

I did not check the Kubernetes or service-name discovery classes, which build their model info from the
engine rather than from index-aligned lists.

Dominant language
Python
Stars
2.6k
Forks
503
Avg merge
4d 17h
Merged PRs (30d)
8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from vllm-project/production-stack

All issues in vllm-project/production-stack

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.