vllm-project/semantic-router

research: provision fleets for mixed workload archetype distributions

Open

#2,556 opened on Jul 15, 2026

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Go (699 forks)github user discovery
area/bencharea/model-selectionarea/researchenhancementevaluationhelp wantedoperationspriority/P2roadmap

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Motivation

Extend FleetSim experiments from a single aggregate workload cumulative distribution function (CDF) to mixtures of versioned workload-archetype distributions, so capacity recommendations can be tested against composition changes.

The related paper calls the analytical method FleetOpt. The current repository class is FleetOptimizer; use repository names in code and make the paper-to-code mapping explicit in docs.

Audited upstream baseline (2026-07-15)

  • FleetSim code is organized under src/fleet-sim/fleet_sim/core/, src/fleet-sim/fleet_sim/routing/, src/fleet-sim/fleet_sim/workload/, and src/fleet-sim/fleet_sim/optimizer/.
  • There is no src/fleet-sim/fleet_sim/simulator/ package on current main.
  • FleetSim already provides fleet/pool/request models, CDF/trace inputs, synthetic workloads, FleetOptimizer, and threshold/Pareto analysis.
  • Existing bundled CDFs represent several example workloads, but there is no canonical mixture/archetype scenario contract.

Research scope

  1. Define a versioned archetype distribution schema: token-demand CDF or trace, arrival process, SLO class, model eligibility, locality/residency constraints, and weight.
  2. Build deterministic mixture sampling and time-varying composition scenarios.
  3. Validate that mixtures reproduce their component/aggregate distributions and preserve correlations explicitly modeled.
  4. Extend FleetOptimizer scenario sweeps to evaluate robust capacity across mixture weights, bursts, and drift.
  5. Compare optimizing for the aggregate CDF, individual archetypes, nominal mixture, and robust/worst-case mixture.
  6. Emit recommendation, objective values, uncertainty/sensitivity, and infeasibility diagnostics.
  7. Keep production actuation separate; this issue provides simulation/evaluation evidence.

Boundaries

  • Do not invent a parallel simulator package or duplicate existing core/routing/workload responsibilities.
  • Archetypes must use privacy-safe aggregate data by default.
  • A mixture of marginal CDFs does not preserve unknown cross-feature correlations; limitations must be stated.
  • Capacity recommendations do not alter semantic model selection on the request path.
  • Hard residency/safety/model-compatibility constraints remain hard.

Acceptance criteria

  • Schema and deterministic fixtures cover fixed and time-varying mixtures.
  • Distribution-validation tests catch weight, normalization, and sampling errors.
  • Baselines and sensitivity to mixture misspecification are reported.
  • Infeasible/SLO-violating scenarios fail explicitly.
  • Existing single-CDF workflows remain compatible.
  • CLI/API/docs use FleetOptimizer and correct current paths.
  • Reproducible benchmark scenarios include composition drift and burst cases.

Likely change surfaces

src/fleet-sim/fleet_sim/workload/, src/fleet-sim/fleet_sim/core/, src/fleet-sim/fleet_sim/routing/, src/fleet-sim/fleet_sim/optimizer/, API/CLI layers, fixtures, docs, and tests.

Related: #2359, #2550, #2554, #2557.

Validation entrypoint

make agent-report ENV=cpu CHANGED_FILES="<space-separated changed files>"

Follow the reported gates and affected E2E profiles.

Contributor guide