bug: stream bandwidth counts old traffic again when its baseline is lost

Đang mở
  • Có gì không ổn?
#14,103 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 2 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
45/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
lua, nginx
Lĩnh vực
backend, observability

Hướng nghiên cứu

Start by reading publish_stream_bytes and the function-level LuaJIT test linked in the issue; the issue does not name source or test file paths. Reproduce the missing-baseline case, then investigate how shared dictionary storage, counters, reloads, and resets interact. Done means regression tests cover first-byte counting, repeated baseline loss with a surviving counter, and full storage, with failures logged and possible missed bytes documented.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Current Behavior

With #14101 applied, apisix_stream_bandwidth can count the same bytes again if its saved baseline is lost but the Prometheus counter survives.

APISIX keeps two values for each stream metrics slot and byte direction:

  • The stream metrics zone stores the total bytes transferred.
  • A baseline in the prometheus-metrics shared dictionary records how many of those bytes have already been added to the Prometheus counter.

Each collection adds the difference between the zone total and the baseline. #14101 treats a missing baseline as zero so that a new slot's first bytes are counted.

However, an existing slot can also lose its baseline if the shared dictionary runs out of space and evicts it. If the counter survives, already-counted bytes are added again:

State Zone total Baseline Prometheus counter
After counting 5 bytes 5 5 5
Baseline is evicted; no new traffic 5 Missing 5
After the next collection 5 5 10

Losing the baseline again can repeat the overcount.

Before #14101, APISIX saved a new baseline without adding any bytes when the baseline was missing. That avoided this overcount, but also skipped a new slot's first bytes. The fix should preserve first-byte counting.

Expected Behavior

Count a new slot's first bytes without counting old traffic again when an existing baseline is lost.

If storage is full and correct accounting is not possible, skip the update and log the failure. Document any bytes that may be missed as a result.

Validation So Far

A function-level LuaJIT test used the actual publish_stream_bytes function with dictionary and counter substitutes. With the zone total fixed at 15, deleting the baseline twice made the counter increase from 15 to 30 to 45.

This confirms what happens when a baseline is missing. It does not reproduce eviction under memory pressure on a running gateway, or show that overcounting happens automatically on every collection. No gateway reproduction logs are available yet.

Proposed Gateway Reproduction

These steps have not been run yet. Use an isolated test gateway.

  1. Apply #14101. Enable stream Prometheus metrics and apisix_stream_metrics_zone. Set nginx_config.meta.lua_shared_dict.prometheus-metrics to a small size, such as 1m, and create a stream route bound to a service.
  2. Send 5 downstream-ingress bytes. Wait for collection and confirm that the target slot's zone total, baseline, and counter are all 5.
  3. Use a test-only Lua probe to fill ngx.shared["prometheus-metrics"] until writes report forced eviction. Verify that the target baseline is gone but its counter remains. If both are gone, the required condition has not been reproduced. Use get_keys(0) to check key presence without refreshing the target entries' LRU positions.
  4. Send no more traffic. After the next collection, check whether the counter rises to 10 while the zone total stays at 5. The counter must survive the collection for this check to be valid.
  5. Repeat the baseline eviction while preserving the counter to check whether the same bytes are counted again.
Possible Fix

Store baselines in a separate shared dictionary and use non-evicting writes such as safe_set. A separate dictionary alone is not enough: its writes must not evict existing baselines. If a write fails, skip the counter update and log the failure.

The design also needs to keep baselines and counters consistent across reloads and resets. Another option is to retain reliable state that distinguishes a new slot from one whose baseline was lost.

Regression tests should cover:

  • Counting a new slot's first bytes.
  • Repeated baseline loss while the counter survives and no new traffic arrives.
  • Full baseline storage.
Environment
  • APISIX: master with #14101 applied.
  • Component: stream Prometheus bandwidth metrics backed by apisix_stream_metrics_zone.
Ngôn ngữ chính
Lua
Star
17.3k
Fork
3k
Merge trung bình
4 ngày 14 giờ
Pull request đã merge (30 ngày)
73

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/apisix

Tất cả issue của apache/apisix

Issue tương tự

Thêm issue về Lua

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.