research: evaluate energy-aware backend/LB policy from FleetSim power models

Motivation

Evaluate energy as an additional serving/LB objective while preserving quality, safety, residency, capacity, and latency constraints. The semantic router continues to choose a preferred logical model per query; an eligible backend/pool is selected below it.

Audited upstream baseline (2026-07-15)

Energy support already exists in FleetSim:

• optimizer/tpw.py provides per-pool and fleet-level analysis, including fleet_tpw_analysis. Its tokens_per_watt field is output-token throughput (tokens/s) divided by watts and is therefore dimensionally output tokens per joule; the historical field/print label must not be read as a different physical unit.
• GPU profiles contain idle/nominal and optional logistic power models.
• optimizer/grid_flex.py models power-versus-P99 trade-offs under concurrency caps.
• Power-model quality is explicit; the code warns that power/throughput are model-dependent and that some profiles are projections only.

The missing capability is trustworthy runtime/backend observation and a downstream LB policy that maps measured/calibrated energy evidence to eligible endpoints. It is not a missing semantic-routing objective implementation.

Research scope

1. Define a backend energy observation contract:
   • backend/pool, hardware, logical and deployable model/version;
   • time window and utilization/concurrency;
   • measured energy/power source and quality, or modeled estimate with calibration version;
   • output tokens/useful completions, latency/SLO, and uncertainty.
2. Calibrate FleetSim power/throughput profiles against real deployment measurements; keep measured and modeled values distinguishable.
3. Evaluate objectives such as joules per successful request or per useful output token, not tok/J in isolation.
4. Let the semantic router emit the preferred logical model and hard constraints. Ordered logical-model fallback is not first-class on current main and, if added through #2294, must be versioned separately.
5. Let the LB/serving layer choose among compatible endpoints using energy, queue, health, capacity, and SLO evidence.
6. Compare energy-oblivious least-loaded/spillover, static efficient pool, and energy-aware policies in FleetSim and shadow telemetry.
7. Add grid/carbon intensity only as a separate versioned input with locality/residency and freshness constraints.
8. Begin with offline evaluation and advise-only recommendations; automatic actuation requires a downstream owner, acknowledgement, rollback, and rate limits.

Architectural and scientific boundaries

• Follow #2513: no capacity/energy solver in semantic ext_proc/model selection.
• Energy cannot introduce a model/backend outside the router's eligible set.
• Authorization, safety, data residency, context compatibility, capacity, and SLO are hard constraints.
• Do not compare the tokens_per_watt/tok/J metric across model sizes without controlling for quality and useful work.
• Do not present projected LOW-quality power profiles as measured production facts.
• Stale/missing telemetry must fall back to the downstream layer's documented energy-oblivious LB policy; implementing that fallback is part of the integration contract, not an existing semantic-router guarantee.

Acceptance criteria

• Observation schema distinguishes measured, calibrated, and modeled energy with provenance/uncertainty.
• Model/hardware/runtime/config versions prevent invalid cross-profile comparison.
• FleetSim experiment reports energy, quality/task success, latency/SLO, cost, and capacity together.
• Simple LB baselines and sensitivity to power-model error are reported.
• Downstream recommendation, acknowledgement, actuation, and measured outcome are separate events.
• Shadow mode proves no hard-constraint or SLO regression before canary.
• Failure, stale data, oscillation, rollback, and partial-deployment tests exist.
• Documentation states exactly which values are measured versus modeled.

Likely change surfaces

src/fleet-sim/fleet_sim/optimizer/tpw.py, src/fleet-sim/fleet_sim/optimizer/grid_flex.py, src/fleet-sim/fleet_sim/gpu_profiles/, src/fleet-sim/fleet_sim/routing/, router/backend observability export, deployment LB/GIE integration, dashboards, docs, and tests.

Related: #2294, #2332, #2359, #2513, #2550, #2551, #2556.

Validation entrypoint

make agent-report ENV=cpu CHANGED_FILES="<space-separated changed files>"

Follow the reported gates and affected E2E profiles.