vllm-project/vllm

[Bug]: Add tests for all parallel sampling parameter combinations

Open

#21,948 opened on Jul 30, 2025

View on GitHub
 (7 comments) (0 reactions) (1 assignee)Python (80,034 stars) (16,816 forks)batch import
buggood first issuekeep-openstale

Description

Verify that we have tests that explicitly exercise parallel sampling (n>1 request sampling parameter) for all of the following combinations:

  • Via AsyncLLM.generate, via LLMEngine add_request() / step()
  • For output_kind equal to each of CUMULATIVE, DELTA, FINAL_ONLY

Ideally a test for each of AsyncLLM and LLMEngine in the appropriate files, with output_kind parameterized.

Additionally we can still test LLM.generate() but this enforces FINAL_ONLY. There is already a test for this one here, but I'm not sure about the other cases.

@sethkimmel3 has reported that the LLMEngine + CUMULATIVE combination is not working properly, this should be exposed and fixed if so.

Contributor guide

Tech stack
python
Domain
backendtesting
Issue type
test
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
blocked
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Pythonpytest
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50
Research direction
Examine the existing test at tests/v1/engine/test llm engine.py around line 120 to understand the current test for LLM.generate. Then identify the test files for AsyncLLM (likely in tests/v1/engine/test async llm.py or similar). Write parameterized tests covering all combinations of n>1 with output kind values CUMULATIVE, DELTA, and FINAL ONLY for both AsyncLLM.generate and LLMEngine add request/step. Additionally, verify the reported bug with LLMEngine + CUMULATIVE and fix if necessary. Check existing tests to avoid duplication.
[Bug]: Add tests for all parallel sampling parameter combinations · vllm-project/vllm#21948 | Good First Issue