When several YAML files each define pipeline.outputs, outputs from files loaded before the main pipeline file are silently dropped

Abierto
  • ¿Algo no está bien?
#12,724 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
55/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
c, yaml
Área
backend, data-engineering

Línea de trabajo

Start by tracing how included YAML files are loaded and how their pipeline.outputs lists are merged; the report identifies glob load order as the trigger. Reproduce with extra-output.yml before main.yml, then reverse the order and compare active outputs through /api/v1/metrics. Done when outputs from both files remain active regardless of load order and the regression is covered by a test.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

status: waiting-for-triage

Bug Report

I split Fluent Bit YAML into several files in one folder. The main file loads them with a glob:

includes:
  - conf/apps/*.yml

More than one of those files has its own pipeline.outputs list.

Fluent Bit loads the glob in filename order (alphabetical) and merges everything into one pipeline. In 5.1.2 we see this:

  • The file that also defines pipeline.inputs (the “main” pipeline) keeps its outputs.
  • Any other file that only (or also) defines pipeline.outputs, but is loaded before that main file, has those outputs silently dropped.
  • There is no error / warning message observ. Config test can still pass.
  • Files that only add filters still work even if they load first.
  • It is two (or more) pipeline.outputs blocks, and the early ones never become live plugins if they are parsed before the main pipeline file.

Typical files:

  • extra-output.yml — pipeline.outputs only (second destination: kafka or extra stdout). Name starts with E, so it loads first.
  • filters.yml — pipeline.filters only.
  • main.yml — pipeline.inputs and pipeline.outputs (primary destination). Name starts with M, so it loads after extra.
  • Both extra-output.yml and main.yml define pipeline.outputs. That is two output lists for one process.

Expected: merge appends all outputs (main plus extra), like filters.

Actual: only main.yml outputs run. Extra file outputs never appear in logs or /api/v1/metrics.

Proof it is load order, not the extra YAML: rename extra-output.yml to zz-extra-output.yml so it loads after main.yml. Restart. Both output lists are live. Same content, only the filename (order) changed. Extra outputs show up as later plugin ids (for example kafka.5 while main still has loki.0 or stdout.0), so they were appended after the main list, not a full replace of main.

Extra loads before main → extra outputs are gone.
Extra loads after main → both lists exist.

How to reproduce

fluent-bit.yml
conf/apps/extra-output.yml   # has pipeline.outputs — sorts BEFORE main.yml
conf/apps/main.yml           # has pipeline.inputs AND pipeline.outputs
fluent-bit.yml:

service:
  flush: 1
  log_level: info
  http_server: on
  http_listen: 127.0.0.1
  http_port: 2020
includes:
  - conf/apps/*.yml

conf/apps/extra-output.yml (second file with outputs, no inputs):

pipeline:
  outputs:
    - name: stdout
      match: "demo"
      alias: extra_stdout

conf/apps/main.yml (inputs plus its own outputs):

pipeline:
  inputs:
    - name: dummy
      dummy: '{"msg":"hello"}'
      tag: demo
      rate: 1
  outputs:
    - name: stdout
      match: "demo"
      alias: main_stdout

fluent-bit -c fluent-bit.yml
curl -sS http://127.0.0.1:2020/api/v1/metrics

Bug: metrics show only main_stdout. Dummy prints once. No extra_stdout. No error.

mv conf/apps/extra-output.yml conf/apps/zz-extra-output.yml
# restart fluent-bit

After rename: main_stdout and extra_stdout. Dummy prints twice.

Same with a kafka output in the early file (plugin in the binary, YAML valid): no kafka until that file sorted after main.yml and the process restarted.

Environment

Fluent Bit 5.1.2, Linux, YAML, HTTP API on port 2020.

Docs gap

The Includes page shows a literal list of files, not *.yml, and not what happens if two files both set pipeline.outputs. Pipeline docs say multiple pipeline sections merge into one pipeline. They do not say early output lists can disappear.

Questions

  • Is this a known bug? If yes, please link the issue
  • what should users do today (5.1.x) that fluent bit support ?
  • Can Fluent Bit keep / concatenate pipeline.outputs from every included file, in any order?

Thanks!!

Lenguaje dominante
C
Estrellas
8.2k
Forks
2k
Merge medio
3 d 20 h
PR fusionados (30 d)
99

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de fluent/fluent-bit

Todos los issues de fluent/fluent-bit

Issues similares

Más issues de C

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.