"cost-optimized" routing strategy never looks at cost (or latency)

Open
  • Is something wrong?
#3,160 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
38/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
api, backend, performance

Research direction

Start with internal/routing/costopt.go and compare its Select implementation with internal/routing/latency.go and the sibling LeastLatencyStrategy. Read internal/routing/target.go and internal/modeldb/litellm.go, then run go build ./... to understand the existing data flow. Done means cost-optimized routing uses real per-token pricing while respecting the existing latency/SLO signal instead of only static priority.

Written by the indexing model from the issue text.

Description

bug
What happened?

The cost-optimized routing strategy doesn't use cost data at all, despite the name and the
documented behavior ("Route to the cheapest provider that meets your SLO" — README.md's routing
strategies table).

CostOptimizedStrategy.Select() (internal/routing/costopt.go) only looks at each target's
static, manually-configured Priority int (internal/routing/target.go) and round-robins within
whichever targets share the lowest one. It never touches internal/modeldb, which already parses
real per-model, per-provider pricing (InputPerToken/OutputPerToken, from a LiteLLM-sourced
pricing table) — that data exists and is used elsewhere, just not here.

It also never touches the SLO half of the claim: every Strategy.Select(targets, tracker *LatencyTracker) implementation receives a live EWMA-latency tracker (internal/routing/ latency.go), and costopt.go takes it as _ and discards it. The sibling LeastLatencyStrategy
in the same package shows the intended usage pattern (tracker.Get(providerID)) that
cost-optimized never calls.

Net effect: default_strategy: cost-optimized in the gateway config produces a priority-tier
round robin with zero live cost or latency signal — not "cheapest provider that meets your SLO."

Steps to reproduce
  1. Configure two targets for the same model with different real per-token pricing but an inverted
    priority (cheaper provider given the higher priority number): 
    routing:
  default_strategy: cost-optimized
  targets:
    my-model:
      - provider: expensive-provider   # e.g. a $30/1M-token model
        priority: 0
      - provider: cheap-provider       # e.g. a $0.50/1M-token model
        priority: 1
  2. Send requests for my-model through the gateway.
  3. Every request routes to expensive-provider (priority 0), regardless of the real price
    difference.

I verified this directly rather than just from reading the source: calling
CostOptimizedStrategy.Select() 200 times against exactly this two-target setup sent all 200
requests to the priority: 0 target and 0 to the other, regardless of which one is actually
cheaper — the function has no path by which a real price difference could change the outcome.

Expected behavior

The strategy should select among healthy targets using their actual per-token cost (via
modeldb's pricing data, estimated against the request's token count) subject to a latency/SLO
constraint (via the existing LatencyTracker) — not a static, manually-set priority number.

Deployment

Self-hosted (Docker Compose)

Version

dev (2026-09-29)

Environment

N/A — verified directly against internal/routing: built the module clean (go build ./..., Go 1.27.1) and ran CostOptimizedStrategy.Select() 200 times against a {expensive: priority 0, cheap: priority 1} target set; 200/200 selections went to expensive, 0/200 to cheap, regardless of any cost signal. Also confirmed by reading internal/modeldb/litellm.go that the pricing data costopt.go never touches does exist and is already parsed elsewhere in the binary.

Relevant logs
Checks
  • I searched existing issues and didn't find a duplicate
  • I redacted any secrets, PII, or customer data from logs/screenshots
Dominant language
Python
Stars
2k
Forks
628
Avg merge
2d 2h
Merged PRs (30d)
259

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from future-agi/future-agi

All issues in future-agi/future-agi

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.