bug: stream bandwidth counts old traffic again when its baseline is lost

Abierto
  • ¿Algo no está bien?
#14,103 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
lua, nginx
Área
backend, observability

Línea de trabajo

Start by reading publish_stream_bytes and the function-level LuaJIT test linked in the issue; the issue does not name source or test file paths. Reproduce the missing-baseline case, then investigate how shared dictionary storage, counters, reloads, and resets interact. Done means regression tests cover first-byte counting, repeated baseline loss with a surviving counter, and full storage, with failures logged and possible missed bytes documented.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Current Behavior

With #14101 applied, apisix_stream_bandwidth can count the same bytes again if its saved baseline is lost but the Prometheus counter survives.

APISIX keeps two values for each stream metrics slot and byte direction:

  • The stream metrics zone stores the total bytes transferred.
  • A baseline in the prometheus-metrics shared dictionary records how many of those bytes have already been added to the Prometheus counter.

Each collection adds the difference between the zone total and the baseline. #14101 treats a missing baseline as zero so that a new slot's first bytes are counted.

However, an existing slot can also lose its baseline if the shared dictionary runs out of space and evicts it. If the counter survives, already-counted bytes are added again:

State Zone total Baseline Prometheus counter
After counting 5 bytes 5 5 5
Baseline is evicted; no new traffic 5 Missing 5
After the next collection 5 5 10

Losing the baseline again can repeat the overcount.

Before #14101, APISIX saved a new baseline without adding any bytes when the baseline was missing. That avoided this overcount, but also skipped a new slot's first bytes. The fix should preserve first-byte counting.

Expected Behavior

Count a new slot's first bytes without counting old traffic again when an existing baseline is lost.

If storage is full and correct accounting is not possible, skip the update and log the failure. Document any bytes that may be missed as a result.

Validation So Far

A function-level LuaJIT test used the actual publish_stream_bytes function with dictionary and counter substitutes. With the zone total fixed at 15, deleting the baseline twice made the counter increase from 15 to 30 to 45.

This confirms what happens when a baseline is missing. It does not reproduce eviction under memory pressure on a running gateway, or show that overcounting happens automatically on every collection. No gateway reproduction logs are available yet.

Proposed Gateway Reproduction

These steps have not been run yet. Use an isolated test gateway.

  1. Apply #14101. Enable stream Prometheus metrics and apisix_stream_metrics_zone. Set nginx_config.meta.lua_shared_dict.prometheus-metrics to a small size, such as 1m, and create a stream route bound to a service.
  2. Send 5 downstream-ingress bytes. Wait for collection and confirm that the target slot's zone total, baseline, and counter are all 5.
  3. Use a test-only Lua probe to fill ngx.shared["prometheus-metrics"] until writes report forced eviction. Verify that the target baseline is gone but its counter remains. If both are gone, the required condition has not been reproduced. Use get_keys(0) to check key presence without refreshing the target entries' LRU positions.
  4. Send no more traffic. After the next collection, check whether the counter rises to 10 while the zone total stays at 5. The counter must survive the collection for this check to be valid.
  5. Repeat the baseline eviction while preserving the counter to check whether the same bytes are counted again.
Possible Fix

Store baselines in a separate shared dictionary and use non-evicting writes such as safe_set. A separate dictionary alone is not enough: its writes must not evict existing baselines. If a write fails, skip the counter update and log the failure.

The design also needs to keep baselines and counters consistent across reloads and resets. Another option is to retain reliable state that distinguishes a new slot from one whose baseline was lost.

Regression tests should cover:

  • Counting a new slot's first bytes.
  • Repeated baseline loss while the counter survives and no new traffic arrives.
  • Full baseline storage.
Environment
  • APISIX: master with #14101 applied.
  • Component: stream Prometheus bandwidth metrics backed by apisix_stream_metrics_zone.
Lenguaje dominante
Lua
Estrellas
17.3k
Forks
3k
Merge medio
4 d 14 h
PR fusionados (30 d)
73

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/apisix

Todos los issues de apache/apisix

Issues similares

Más issues de Lua

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.