Fluent Bit 5.1.3: Accumulated DLQ chunks exhaust storage.max_chunks_up and block the OTLP output pipeline

Aperta
  • Qualcosa non va?
#12,574 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
c, docker
Ambito
observability, stream-processing

Direzione di ricerca

Start by reproducing with the YAML config in the report: enable storage.keep.rejected and storage.max_chunks_up, point the opentelemetry output at a mock receiver returning 500, then watch curl localhost:2020/api/v1/storage for fs_chunks_up reaching the limit. Next, check whether files in the rejected directory are counted against storage.max_chunks_up after a restart. Done means new records flush while rejected chunks remain on disk, and a regression test covers that case.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

status: waiting-for-triage

Bug Report

Describe the bug

When Dead Letter Queue (DLQ) is enabled with filesystem buffering, accumulated rejected chunks appear to consume the available storage.max_chunks_up capacity.

Once fs_chunks_up reaches the configured limit, Fluent Bit stops forwarding new records to the OpenTelemetry output, while the OpenTelemetry input continues accepting incoming requests and returning HTTP 201 Created.

The output pipeline remains blocked even though new records are accepted and filesystem buffering is enabled.

The issue was reproduced with storage.max_chunks_up values of 128 and 256.

Increasing the limit from 128 to 256 temporarily restored output processing. However, after accumulating more DLQ chunks and restarting Fluent Bit with the existing storage volume, the issue occurred again.

To Reproduce

Steps to reproduce the problem:

The reproduction environment uses Fluent Bit 5.1.3 with an OpenTelemetry input, OpenTelemetry output, filesystem buffering, and a mock OTLP receiver that returns HTTP 400 or 500.

1. Configure Fluent Bit

service:
  flush: 1
  log_level: debug
  scheduler.base: 1
  scheduler.cap: 2
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  storage.path: /fluent-bit/storage
  storage.sync: normal
  storage.max_chunks_up: 256
  storage.keep.rejected: on
  storage.rejected.path: rejected

pipeline:
  inputs:
    - name: opentelemetry
      listen: 0.0.0.0
      port: 4318
      tag: dlq.test
      tag_from_uri: false
      storage.type: filesystem

  outputs:
    - name: opentelemetry
      match: "*"
      host: mock-otel
      port: 8090
      logs_uri: /v1/logs
      tls: off
      retry_limit: ${RETRY_LIMIT}
      log_response_payload: on

2. Start the reproduction environment

docker compose up -d --build

3. Send OTLP log records repeatedly

curl -i \
  -H "Content-Type: application/json" \
  --data-binary @test-record.json \
  http://localhost:4318/v1/logs

The mock OTLP receiver returns HTTP 400 or 500, causing rejected chunks to accumulate in:

/fluent-bit/storage/rejected

4. Accumulate rejected chunks until the storage limit is reached

Monitor the storage metrics:

curl -s localhost:2020/api/v1/storage |
  jq '.storage_layer.chunks'

With storage.max_chunks_up: 256, more than 250 DLQ files were accumulated.

5. Observe the pipeline becoming blocked (with or without restart)

The issue can occur in two scenarios:

  • Without restarting Fluent Bit: As rejected chunks accumulate, the output pipeline may stop processing new records once fs_chunks_up reaches the configured limit.
  • After restarting Fluent Bit: In some tests, the pipeline continued processing until Fluent Bit was restarted with the existing DLQ files. After restart, fs_chunks_up immediately reached the configured limit, and the output pipeline became blocked.

To reproduce the second scenario, restart Fluent Bit without deleting the storage volume:

docker compose restart fluent-bit

The following storage metrics were observed after restart:

{
  "total_chunks": 260,
  "fs_chunks": 260,
  "fs_chunks_up": 256,
  "fs_chunks_down": 4
}

The configured storage.max_chunks_up limit was fully occupied immediately after startup.

Note: A restart is not always required to reproduce the issue. The pipeline can become blocked during normal operation as DLQ chunks accumulate.

6. Send another OTLP record

The OpenTelemetry input still responds with HTTP 201 Created.

The input metrics increase:

input.records: 11 -> 12
input.bytes:   3124 -> 3408

However, the output metrics remain unchanged:

proc_records:    0
errors:          0
retries:         0
dropped_records: 0

No new requests reach the mock OTLP receiver.

Expected behavior

Persisted DLQ chunks should not indefinitely block the active output pipeline.

Even when a large number of rejected chunks have accumulated, Fluent Bit should be able to continue processing and flushing new filesystem-buffered records.

Ideally, DLQ storage should not consume all available UP chunk capacity required by active pipeline processing.

Your Environment
  • Version used: Fluent Bit 5.1.3
  • Configuration: YAML configuration above
  • Environment name and version: Local container environment using docker compose
  • Server type and version: Fluent Bit container and a Python ThreadingHTTPServer mock receiver
  • Operating System and version: macOS host with a Linux container runtime
  • Filters and plugins:
    • Input: opentelemetry
    • Output: opentelemetry
    • Storage: filesystem
    • DLQ: storage.keep.rejected: on
    • Filters: none
Lingua principale
C
Stelle
8.2k
Fork
2k
Merge medio
3g 20h
PR unite (30g)
99

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di fluent/fluent-bit

Tutte le issue di fluent/fluent-bit

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.