Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

No long-term view of inference server performance to compare against current performance

Abierto
  • ¿Algo no está bien?
#1,466 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go
Área
observability

Línea de trabajo

Read core/cost/usage/usage.go, core/cost/ledger/row.go, writer.go, and query.go, then trace archive replay in cmd/cortex/archive.go. First decide what durable performance data and baseline the feature should use, and how long-window results should be bucketed. Done means performance history can be compared across time and distinguished by server and model; the issue leaves those design choices open.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement laptop

Summary

To decide whether to switch inference servers, you need to know whether the one you are on is slower or less reliable than it usually is. That needs two things side by side: how it is doing now, and how it did over the last week or month. Cortex can show the first, roughly (see #1465 for why "roughly"), but not the second. It keeps no long-term record of how a server performed, only of what it cost.

The problem

  1. Latency is gone after 6 hours. Response times live only in the in-memory usage ring: 360 one-minute buckets (usage.NumBuckets, BucketWidth). window=24h is refused with window 24h0m0s exceeds the 6h0m0s retained. After a restart, the ring refills only from the session archive's last six hours (usageReplayBudget, cmd/cortex/archive.go).

  2. The durable store records cost, not performance. The cost ledger (core/cost/ledger) keeps per-minute rows on disk for 31 days, keyed by endpoint, model and agent. Its rows hold usage.Counts: requests, errors and token counts. They hold no latency, because latency is kept beside Counts in the ring bucket (latSum, latSumSq, latN), not inside it. A 7d window on a live proxy returns latMeanMs 0 for every endpoint.

  3. Long windows come back as one number, not a trend. today, 7d and month are served from the ledger as a single bucket covering the whole window (bucketSeconds: 604800 for 7d, whatever resolution is asked for). So even the error rate, which the ledger does keep, cannot be plotted over days. Only those three windows exist: 30d is refused, and nothing lies between 6h and today.

  4. The errors kept on disk cannot be told apart. The ledger stores Errors as a count, without the status. A 429 (rate limited: wait, or switch) and a 5xx (server broken: switch) call for different responses, and the ledger merges them. Only the 6-hour ring has group=status.

  5. The latency we do record measures the wrong thing for this question. SessionEvent.Duration is total wall time, from the request entering the listener to the response being recorded. It grows with how long the answer is and includes Cortex's own plugin time. A server that has slowed down and an agent that asked for a longer answer look the same. There is no time-to-first-token, and no decode speed (tokens per second) (#951), and only a mean with standard deviation, with no percentiles (#1026).

  6. History is not tied to a server. Ledger rows are keyed by endpoint (host), not by router server name, and a server's URL can change. The same mapping gap as #1465 applies here, over a longer span.

The raw material does exist: the session archive keeps 30 days of events on disk, each with durationMs, host, status and token counts. But only its last six hours are ever replayed, and nothing aggregates it beyond that.

What the live proxy can and cannot say today

Last 7 days from the ledger (/v1/usage?window=7d&group=endpoint):

server requests errors latency over the week trend over days
ete-int 9711 368 (3.8%) not stored not available
glm 6183 153 (2.5%) not stored not available

Set against the 6-hour view in #1465 (glm at 50% errors over 6 requests), this is exactly the comparison the feature is for. The data needed to make it is either missing (latency) or flattened (one total per week).

Questions for the fix

  • What performance data should be durable: latency sum, sum of squares and count per row (cheap; matches the ring), or a histogram, which percentiles (#1026) need?
  • Should that data live in the cost ledger, which is already per-minute, keyed by endpoint and model, and kept 31 days, or in a separate store? The ledger currently admits inference traffic only, which is the right scope for this.
  • Should long windows be plotted as a series (for example hourly buckets for 7 days, daily for a month) instead of one total?
  • What is the baseline: the same server over the last 7 or 30 days, or that server and model at the same hour of day?
  • How does this relate to #1400, which proposes alerting from a per-(endpoint, model) baseline? That issue needs the same durable data. This one adds a view where a person makes the comparison themselves. One store should serve both.
  • Should a baseline be per server or per (server, model)? A slower model on the same server is not a degraded server.

Pointers

  • In-memory ring: core/cost/usage/usage.go (bucket, foldInto, NumBuckets).
  • Ledger: core/cost/ledger/row.go (Row, its key), writer.go (Record's inference-only admission, Errors), query.go (single-bucket answers).
  • Archive replay: cmd/cortex/archive.go (replayUsage, usageReplayBudget); archive retention is in core/session/archive (30 days).
  • Event timing: pipeline.SessionEvent.Duration.
  • Related: #1465 (the pane mixes servers), #1400 (performance alerts), #951 (TTFT and response-time metrics), #1026 (percentiles).

Assisted-By: Claude (Anthropic AI) [email protected]

Lenguaje dominante
Go
Estrellas
18
Forks
42
Merge medio
12 h 31 min
PR fusionados (30 d)
258

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de rossoctl/cortex

Todos los issues de rossoctl/cortex

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.