bug: PriorityRouter always picks the first engine when several engines have the same lowest load

Open Beginner friendly
  • Is something wrong?
#1,203 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 5 days

@Scanf-s is already working on this.

Since Nov 12, 2026.

  • #1204 by @Scanf-s — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend, distributed-systems

Research direction

Read the PriorityRouter implementation and compare its tie handling with PrefixAwareRouter, which the issue says already uses random choice. Run the tests in src/tests/test_priority_router.py and add coverage for multiple engines sharing the lowest load. Done means equally loaded engines can all be selected instead of always favoring the URL that sorts first.

Written by the indexing model from the issue text.

Description

bug
Describe the bug

PriorityRouter sends a request whose priority is below the priority_threshold to the least-loaded engine.
But it picks the first engine URLs because of the sorted() and min() function.
When it comes to the several engines have the same lowest load, it always picks the engine whose URL sorts first.

For example, when all engines are idle or every engine has N request, the first engine gets more requests than the other engines.

To Reproduce

Router version: lmcache/lmstack-router:v0.1.14 and the same code on main branch(405a2fa).

  1. Please start the router with --routing-logic priority --priority-default 0 --priority-threshold 1 in front of the several engines that serve the same model.

  2. Send requests without a priority to make every request uses the least-loaded path.

  3. Count the requests that each engine receives

(In our setup, 4 vLLM engines, 30 minutes of VLM requests), the engines got 298, 276, 256 and 252 requests, in the order of their URLs.)

Expected behavior

I hope the router spreads the reuqests over all engiens that have the same lowest load.

Additional context

In my point of view, there are two simple ways to break the tie.

  1. Round-robin approach
  2. random.choice (PrefixAwareRouter already does this)

I will open a PR with option 2 to follow the existing concept with several tests in src/tests/test_priority_router.py

Dominant language
Python
Stars
2.7k
Forks
558
Avg merge
4d 23h
Merged PRs (30d)
5

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from vllm-project/production-stack

All issues in vllm-project/production-stack

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.