Fluent Bit 5.1.3: Accumulated DLQ chunks exhaust storage.max_chunks_up and block the OTLP output pipeline

Đang mở
  • Có gì không ổn?
#12,574 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
25/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
c, docker
Lĩnh vực
observability, stream-processing

Hướng nghiên cứu

Start by reproducing with the YAML config in the report: enable storage.keep.rejected and storage.max_chunks_up, point the opentelemetry output at a mock receiver returning 500, then watch curl localhost:2020/api/v1/storage for fs_chunks_up reaching the limit. Next, check whether files in the rejected directory are counted against storage.max_chunks_up after a restart. Done means new records flush while rejected chunks remain on disk, and a regression test covers that case.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

status: waiting-for-triage

Bug Report

Describe the bug

When Dead Letter Queue (DLQ) is enabled with filesystem buffering, accumulated rejected chunks appear to consume the available storage.max_chunks_up capacity.

Once fs_chunks_up reaches the configured limit, Fluent Bit stops forwarding new records to the OpenTelemetry output, while the OpenTelemetry input continues accepting incoming requests and returning HTTP 201 Created.

The output pipeline remains blocked even though new records are accepted and filesystem buffering is enabled.

The issue was reproduced with storage.max_chunks_up values of 128 and 256.

Increasing the limit from 128 to 256 temporarily restored output processing. However, after accumulating more DLQ chunks and restarting Fluent Bit with the existing storage volume, the issue occurred again.

To Reproduce

Steps to reproduce the problem:

The reproduction environment uses Fluent Bit 5.1.3 with an OpenTelemetry input, OpenTelemetry output, filesystem buffering, and a mock OTLP receiver that returns HTTP 400 or 500.

1. Configure Fluent Bit

service:
  flush: 1
  log_level: debug
  scheduler.base: 1
  scheduler.cap: 2
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  storage.path: /fluent-bit/storage
  storage.sync: normal
  storage.max_chunks_up: 256
  storage.keep.rejected: on
  storage.rejected.path: rejected

pipeline:
  inputs:
    - name: opentelemetry
      listen: 0.0.0.0
      port: 4318
      tag: dlq.test
      tag_from_uri: false
      storage.type: filesystem

  outputs:
    - name: opentelemetry
      match: "*"
      host: mock-otel
      port: 8090
      logs_uri: /v1/logs
      tls: off
      retry_limit: ${RETRY_LIMIT}
      log_response_payload: on

2. Start the reproduction environment

docker compose up -d --build

3. Send OTLP log records repeatedly

curl -i \
  -H "Content-Type: application/json" \
  --data-binary @test-record.json \
  http://localhost:4318/v1/logs

The mock OTLP receiver returns HTTP 400 or 500, causing rejected chunks to accumulate in:

/fluent-bit/storage/rejected

4. Accumulate rejected chunks until the storage limit is reached

Monitor the storage metrics:

curl -s localhost:2020/api/v1/storage |
  jq '.storage_layer.chunks'

With storage.max_chunks_up: 256, more than 250 DLQ files were accumulated.

5. Observe the pipeline becoming blocked (with or without restart)

The issue can occur in two scenarios:

  • Without restarting Fluent Bit: As rejected chunks accumulate, the output pipeline may stop processing new records once fs_chunks_up reaches the configured limit.
  • After restarting Fluent Bit: In some tests, the pipeline continued processing until Fluent Bit was restarted with the existing DLQ files. After restart, fs_chunks_up immediately reached the configured limit, and the output pipeline became blocked.

To reproduce the second scenario, restart Fluent Bit without deleting the storage volume:

docker compose restart fluent-bit

The following storage metrics were observed after restart:

{
  "total_chunks": 260,
  "fs_chunks": 260,
  "fs_chunks_up": 256,
  "fs_chunks_down": 4
}

The configured storage.max_chunks_up limit was fully occupied immediately after startup.

Note: A restart is not always required to reproduce the issue. The pipeline can become blocked during normal operation as DLQ chunks accumulate.

6. Send another OTLP record

The OpenTelemetry input still responds with HTTP 201 Created.

The input metrics increase:

input.records: 11 -> 12
input.bytes:   3124 -> 3408

However, the output metrics remain unchanged:

proc_records:    0
errors:          0
retries:         0
dropped_records: 0

No new requests reach the mock OTLP receiver.

Expected behavior

Persisted DLQ chunks should not indefinitely block the active output pipeline.

Even when a large number of rejected chunks have accumulated, Fluent Bit should be able to continue processing and flushing new filesystem-buffered records.

Ideally, DLQ storage should not consume all available UP chunk capacity required by active pipeline processing.

Your Environment
  • Version used: Fluent Bit 5.1.3
  • Configuration: YAML configuration above
  • Environment name and version: Local container environment using docker compose
  • Server type and version: Fluent Bit container and a Python ThreadingHTTPServer mock receiver
  • Operating System and version: macOS host with a Linux container runtime
  • Filters and plugins:
    • Input: opentelemetry
    • Output: opentelemetry
    • Storage: filesystem
    • DLQ: storage.keep.rejected: on
    • Filters: none
Ngôn ngữ chính
C
Star
8.2k
Fork
2k
Merge trung bình
3 ngày 20 giờ
Pull request đã merge (30 ngày)
99

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của fluent/fluent-bit

Tất cả issue của fluent/fluent-bit

Issue tương tự

Thêm issue về C

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.