When several YAML files each define pipeline.outputs, outputs from files loaded before the main pipeline file are silently dropped

Aperta
  • Qualcosa non va?
#12,724 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
c, yaml
Ambito
backend, data-engineering

Direzione di ricerca

Start by tracing how included YAML files are loaded and how their pipeline.outputs lists are merged; the report identifies glob load order as the trigger. Reproduce with extra-output.yml before main.yml, then reverse the order and compare active outputs through /api/v1/metrics. Done when outputs from both files remain active regardless of load order and the regression is covered by a test.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

status: waiting-for-triage

Bug Report

I split Fluent Bit YAML into several files in one folder. The main file loads them with a glob:

includes:
  - conf/apps/*.yml

More than one of those files has its own pipeline.outputs list.

Fluent Bit loads the glob in filename order (alphabetical) and merges everything into one pipeline. In 5.1.2 we see this:

  • The file that also defines pipeline.inputs (the “main” pipeline) keeps its outputs.
  • Any other file that only (or also) defines pipeline.outputs, but is loaded before that main file, has those outputs silently dropped.
  • There is no error / warning message observ. Config test can still pass.
  • Files that only add filters still work even if they load first.
  • It is two (or more) pipeline.outputs blocks, and the early ones never become live plugins if they are parsed before the main pipeline file.

Typical files:

  • extra-output.yml — pipeline.outputs only (second destination: kafka or extra stdout). Name starts with E, so it loads first.
  • filters.yml — pipeline.filters only.
  • main.yml — pipeline.inputs and pipeline.outputs (primary destination). Name starts with M, so it loads after extra.
  • Both extra-output.yml and main.yml define pipeline.outputs. That is two output lists for one process.

Expected: merge appends all outputs (main plus extra), like filters.

Actual: only main.yml outputs run. Extra file outputs never appear in logs or /api/v1/metrics.

Proof it is load order, not the extra YAML: rename extra-output.yml to zz-extra-output.yml so it loads after main.yml. Restart. Both output lists are live. Same content, only the filename (order) changed. Extra outputs show up as later plugin ids (for example kafka.5 while main still has loki.0 or stdout.0), so they were appended after the main list, not a full replace of main.

Extra loads before main → extra outputs are gone.
Extra loads after main → both lists exist.

How to reproduce

fluent-bit.yml
conf/apps/extra-output.yml   # has pipeline.outputs — sorts BEFORE main.yml
conf/apps/main.yml           # has pipeline.inputs AND pipeline.outputs
fluent-bit.yml:

service:
  flush: 1
  log_level: info
  http_server: on
  http_listen: 127.0.0.1
  http_port: 2020
includes:
  - conf/apps/*.yml

conf/apps/extra-output.yml (second file with outputs, no inputs):

pipeline:
  outputs:
    - name: stdout
      match: "demo"
      alias: extra_stdout

conf/apps/main.yml (inputs plus its own outputs):

pipeline:
  inputs:
    - name: dummy
      dummy: '{"msg":"hello"}'
      tag: demo
      rate: 1
  outputs:
    - name: stdout
      match: "demo"
      alias: main_stdout

fluent-bit -c fluent-bit.yml
curl -sS http://127.0.0.1:2020/api/v1/metrics

Bug: metrics show only main_stdout. Dummy prints once. No extra_stdout. No error.

mv conf/apps/extra-output.yml conf/apps/zz-extra-output.yml
# restart fluent-bit

After rename: main_stdout and extra_stdout. Dummy prints twice.

Same with a kafka output in the early file (plugin in the binary, YAML valid): no kafka until that file sorted after main.yml and the process restarted.

Environment

Fluent Bit 5.1.2, Linux, YAML, HTTP API on port 2020.

Docs gap

The Includes page shows a literal list of files, not *.yml, and not what happens if two files both set pipeline.outputs. Pipeline docs say multiple pipeline sections merge into one pipeline. They do not say early output lists can disappear.

Questions

  • Is this a known bug? If yes, please link the issue
  • what should users do today (5.1.x) that fluent bit support ?
  • Can Fluent Bit keep / concatenate pipeline.outputs from every included file, in any order?

Thanks!!

Lingua principale
C
Stelle
8.2k
Fork
2k
Merge medio
3g 20h
PR unite (30g)
99

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di fluent/fluent-bit

Tutte le issue di fluent/fluent-bit

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.