bug: stream bandwidth counts old traffic again when its baseline is lost

オープン
  • 何か問題がありますか？
#14,103 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
45/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
lua, nginx
領域
backend, observability

調査の方向性

Start by reading publish_stream_bytes and the function-level LuaJIT test linked in the issue; the issue does not name source or test file paths. Reproduce the missing-baseline case, then investigate how shared dictionary storage, counters, reloads, and resets interact. Done means regression tests cover first-byte counting, repeated baseline loss with a surviving counter, and full storage, with failures logged and possible missed bytes documented.

索引モデルが issue の本文から書いたものです。

説明

Current Behavior

With #14101 applied, apisix_stream_bandwidth can count the same bytes again if its saved baseline is lost but the Prometheus counter survives.

APISIX keeps two values for each stream metrics slot and byte direction:

  • The stream metrics zone stores the total bytes transferred.
  • A baseline in the prometheus-metrics shared dictionary records how many of those bytes have already been added to the Prometheus counter.

Each collection adds the difference between the zone total and the baseline. #14101 treats a missing baseline as zero so that a new slot's first bytes are counted.

However, an existing slot can also lose its baseline if the shared dictionary runs out of space and evicts it. If the counter survives, already-counted bytes are added again:

State Zone total Baseline Prometheus counter
After counting 5 bytes 5 5 5
Baseline is evicted; no new traffic 5 Missing 5
After the next collection 5 5 10

Losing the baseline again can repeat the overcount.

Before #14101, APISIX saved a new baseline without adding any bytes when the baseline was missing. That avoided this overcount, but also skipped a new slot's first bytes. The fix should preserve first-byte counting.

Expected Behavior

Count a new slot's first bytes without counting old traffic again when an existing baseline is lost.

If storage is full and correct accounting is not possible, skip the update and log the failure. Document any bytes that may be missed as a result.

Validation So Far

A function-level LuaJIT test used the actual publish_stream_bytes function with dictionary and counter substitutes. With the zone total fixed at 15, deleting the baseline twice made the counter increase from 15 to 30 to 45.

This confirms what happens when a baseline is missing. It does not reproduce eviction under memory pressure on a running gateway, or show that overcounting happens automatically on every collection. No gateway reproduction logs are available yet.

Proposed Gateway Reproduction

These steps have not been run yet. Use an isolated test gateway.

  1. Apply #14101. Enable stream Prometheus metrics and apisix_stream_metrics_zone. Set nginx_config.meta.lua_shared_dict.prometheus-metrics to a small size, such as 1m, and create a stream route bound to a service.
  2. Send 5 downstream-ingress bytes. Wait for collection and confirm that the target slot's zone total, baseline, and counter are all 5.
  3. Use a test-only Lua probe to fill ngx.shared["prometheus-metrics"] until writes report forced eviction. Verify that the target baseline is gone but its counter remains. If both are gone, the required condition has not been reproduced. Use get_keys(0) to check key presence without refreshing the target entries' LRU positions.
  4. Send no more traffic. After the next collection, check whether the counter rises to 10 while the zone total stays at 5. The counter must survive the collection for this check to be valid.
  5. Repeat the baseline eviction while preserving the counter to check whether the same bytes are counted again.
Possible Fix

Store baselines in a separate shared dictionary and use non-evicting writes such as safe_set. A separate dictionary alone is not enough: its writes must not evict existing baselines. If a write fails, skip the counter update and log the failure.

The design also needs to keep baselines and counters consistent across reloads and resets. Another option is to retain reliable state that distinguishes a new slot from one whose baseline was lost.

Regression tests should cover:

  • Counting a new slot's first bytes.
  • Repeated baseline loss while the counter survives and no new traffic arrives.
  • Full baseline storage.
Environment
  • APISIX: master with #14101 applied.
  • Component: stream Prometheus bandwidth metrics backed by apisix_stream_metrics_zone.
主要言語
Lua
スター
17.3k
フォーク
3k
平均マージ
4日 14時間
マージ済み PR（30日）
73

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/apisix のほかの issue

apache/apisix の issue をすべて見る

似ている issue

Lua の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。