No long-term view of inference server performance to compare against current performance

Read core/cost/usage/usage.go, core/cost/ledger/row.go, writer.go, and query.go, then trace archive replay in cmd/cortex/archive.go. First decide what durable performance data and baseline the feature should use, and how long-window results should be bucketed. Done means performance history can be compared across time and distinguished by server and model; the issue leaves those design choices open.

Summary

To decide whether to switch inference servers, you need to know whether the one you are on is slower or less reliable than it usually is. That needs two things side by side: how it is doing now, and how it did over the last week or month. Cortex can show the first, roughly (see #1465 for why "roughly"), but not the second. It keeps no long-term record of how a server performed, only of what it cost.

The problem

1. Latency is gone after 6 hours. Response times live only in the in-memory usage ring: 360 one-minute buckets (usage.NumBuckets, BucketWidth). window=24h is refused with window 24h0m0s exceeds the 6h0m0s retained. After a restart, the ring refills only from the session archive's last six hours (usageReplayBudget, cmd/cortex/archive.go).

2. The durable store records cost, not performance. The cost ledger (core/cost/ledger) keeps per-minute rows on disk for 31 days, keyed by endpoint, model and agent. Its rows hold usage.Counts: requests, errors and token counts. They hold no latency, because latency is kept beside Counts in the ring bucket (latSum, latSumSq, latN), not inside it. A 7d window on a live proxy returns latMeanMs 0 for every endpoint.

3. Long windows come back as one number, not a trend. today, 7d and month are served from the ledger as a single bucket covering the whole window (bucketSeconds: 604800 for 7d, whatever resolution is asked for). So even the error rate, which the ledger does keep, cannot be plotted over days. Only those three windows exist: 30d is refused, and nothing lies between 6h and today.

4. The errors kept on disk cannot be told apart. The ledger stores Errors as a count, without the status. A 429 (rate limited: wait, or switch) and a 5xx (server broken: switch) call for different responses, and the ledger merges them. Only the 6-hour ring has group=status.

5. The latency we do record measures the wrong thing for this question. SessionEvent.Duration is total wall time, from the request entering the listener to the response being recorded. It grows with how long the answer is and includes Cortex's own plugin time. A server that has slowed down and an agent that asked for a longer answer look the same. There is no time-to-first-token, and no decode speed (tokens per second) (#951), and only a mean with standard deviation, with no percentiles (#1026).

6. History is not tied to a server. Ledger rows are keyed by endpoint (host), not by router server name, and a server's URL can change. The same mapping gap as #1465 applies here, over a longer span.

The raw material does exist: the session archive keeps 30 days of events on disk, each with durationMs, host, status and token counts. But only its last six hours are ever replayed, and nothing aggregates it beyond that.

What the live proxy can and cannot say today

Last 7 days from the ledger (/v1/usage?window=7d&group=endpoint):

Set against the 6-hour view in #1465 (glm at 50% errors over 6 requests), this is exactly the comparison the feature is for. The data needed to make it is either missing (latency) or flattened (one total per week).

Questions for the fix

• What performance data should be durable: latency sum, sum of squares and count per row (cheap; matches the ring), or a histogram, which percentiles (#1026) need?
• Should that data live in the cost ledger, which is already per-minute, keyed by endpoint and model, and kept 31 days, or in a separate store? The ledger currently admits inference traffic only, which is the right scope for this.
• Should long windows be plotted as a series (for example hourly buckets for 7 days, daily for a month) instead of one total?
• What is the baseline: the same server over the last 7 or 30 days, or that server and model at the same hour of day?
• How does this relate to #1400, which proposes alerting from a per-(endpoint, model) baseline? That issue needs the same durable data. This one adds a view where a person makes the comparison themselves. One store should serve both.
• Should a baseline be per server or per (server, model)? A slower model on the same server is not a degraded server.

Pointers

• In-memory ring: core/cost/usage/usage.go (bucket, foldInto, NumBuckets).
• Ledger: core/cost/ledger/row.go (Row, its key), writer.go (Record's inference-only admission, Errors), query.go (single-bucket answers).
• Archive replay: cmd/cortex/archive.go (replayUsage, usageReplayBudget); archive retention is in core/session/archive (30 days).
• Event timing: pipeline.SessionEvent.Duration.
• Related: #1465 (the pane mixes servers), #1400 (performance alerts), #951 (TTFT and response-time metrics), #1026 (percentiles).

Assisted-By: Claude (Anthropic AI) [email protected]