Fluent Bit 5.1.3: Accumulated DLQ chunks exhaust storage.max_chunks_up and block the OTLP output pipeline

Abierto
  • ¿Algo no está bien?
#12,574 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
c, docker
Área
observability, stream-processing

Línea de trabajo

Start by reproducing with the YAML config in the report: enable storage.keep.rejected and storage.max_chunks_up, point the opentelemetry output at a mock receiver returning 500, then watch curl localhost:2020/api/v1/storage for fs_chunks_up reaching the limit. Next, check whether files in the rejected directory are counted against storage.max_chunks_up after a restart. Done means new records flush while rejected chunks remain on disk, and a regression test covers that case.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

status: waiting-for-triage

Bug Report

Describe the bug

When Dead Letter Queue (DLQ) is enabled with filesystem buffering, accumulated rejected chunks appear to consume the available storage.max_chunks_up capacity.

Once fs_chunks_up reaches the configured limit, Fluent Bit stops forwarding new records to the OpenTelemetry output, while the OpenTelemetry input continues accepting incoming requests and returning HTTP 201 Created.

The output pipeline remains blocked even though new records are accepted and filesystem buffering is enabled.

The issue was reproduced with storage.max_chunks_up values of 128 and 256.

Increasing the limit from 128 to 256 temporarily restored output processing. However, after accumulating more DLQ chunks and restarting Fluent Bit with the existing storage volume, the issue occurred again.

To Reproduce

Steps to reproduce the problem:

The reproduction environment uses Fluent Bit 5.1.3 with an OpenTelemetry input, OpenTelemetry output, filesystem buffering, and a mock OTLP receiver that returns HTTP 400 or 500.

1. Configure Fluent Bit

service:
  flush: 1
  log_level: debug
  scheduler.base: 1
  scheduler.cap: 2
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  storage.path: /fluent-bit/storage
  storage.sync: normal
  storage.max_chunks_up: 256
  storage.keep.rejected: on
  storage.rejected.path: rejected

pipeline:
  inputs:
    - name: opentelemetry
      listen: 0.0.0.0
      port: 4318
      tag: dlq.test
      tag_from_uri: false
      storage.type: filesystem

  outputs:
    - name: opentelemetry
      match: "*"
      host: mock-otel
      port: 8090
      logs_uri: /v1/logs
      tls: off
      retry_limit: ${RETRY_LIMIT}
      log_response_payload: on

2. Start the reproduction environment

docker compose up -d --build

3. Send OTLP log records repeatedly

curl -i \
  -H "Content-Type: application/json" \
  --data-binary @test-record.json \
  http://localhost:4318/v1/logs

The mock OTLP receiver returns HTTP 400 or 500, causing rejected chunks to accumulate in:

/fluent-bit/storage/rejected

4. Accumulate rejected chunks until the storage limit is reached

Monitor the storage metrics:

curl -s localhost:2020/api/v1/storage |
  jq '.storage_layer.chunks'

With storage.max_chunks_up: 256, more than 250 DLQ files were accumulated.

5. Observe the pipeline becoming blocked (with or without restart)

The issue can occur in two scenarios:

  • Without restarting Fluent Bit: As rejected chunks accumulate, the output pipeline may stop processing new records once fs_chunks_up reaches the configured limit.
  • After restarting Fluent Bit: In some tests, the pipeline continued processing until Fluent Bit was restarted with the existing DLQ files. After restart, fs_chunks_up immediately reached the configured limit, and the output pipeline became blocked.

To reproduce the second scenario, restart Fluent Bit without deleting the storage volume:

docker compose restart fluent-bit

The following storage metrics were observed after restart:

{
  "total_chunks": 260,
  "fs_chunks": 260,
  "fs_chunks_up": 256,
  "fs_chunks_down": 4
}

The configured storage.max_chunks_up limit was fully occupied immediately after startup.

Note: A restart is not always required to reproduce the issue. The pipeline can become blocked during normal operation as DLQ chunks accumulate.

6. Send another OTLP record

The OpenTelemetry input still responds with HTTP 201 Created.

The input metrics increase:

input.records: 11 -> 12
input.bytes:   3124 -> 3408

However, the output metrics remain unchanged:

proc_records:    0
errors:          0
retries:         0
dropped_records: 0

No new requests reach the mock OTLP receiver.

Expected behavior

Persisted DLQ chunks should not indefinitely block the active output pipeline.

Even when a large number of rejected chunks have accumulated, Fluent Bit should be able to continue processing and flushing new filesystem-buffered records.

Ideally, DLQ storage should not consume all available UP chunk capacity required by active pipeline processing.

Your Environment
  • Version used: Fluent Bit 5.1.3
  • Configuration: YAML configuration above
  • Environment name and version: Local container environment using docker compose
  • Server type and version: Fluent Bit container and a Python ThreadingHTTPServer mock receiver
  • Operating System and version: macOS host with a Linux container runtime
  • Filters and plugins:
    • Input: opentelemetry
    • Output: opentelemetry
    • Storage: filesystem
    • DLQ: storage.keep.rejected: on
    • Filters: none
Lenguaje dominante
C
Estrellas
8.2k
Forks
2k
Merge medio
3 d 20 h
PR fusionados (30 d)
99

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de fluent/fluent-bit

Todos los issues de fluent/fluent-bit

Issues similares

Más issues de C

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.