When several YAML files each define pipeline.outputs, outputs from files loaded before the main pipeline file are silently dropped

オープン
  • 何か問題がありますか？
#12,724 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
55/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
c, yaml
領域
backend, data-engineering

調査の方向性

Start by tracing how included YAML files are loaded and how their pipeline.outputs lists are merged; the report identifies glob load order as the trigger. Reproduce with extra-output.yml before main.yml, then reverse the order and compare active outputs through /api/v1/metrics. Done when outputs from both files remain active regardless of load order and the regression is covered by a test.

索引モデルが issue の本文から書いたものです。

説明

status: waiting-for-triage

Bug Report

I split Fluent Bit YAML into several files in one folder. The main file loads them with a glob:

includes:
  - conf/apps/*.yml

More than one of those files has its own pipeline.outputs list.

Fluent Bit loads the glob in filename order (alphabetical) and merges everything into one pipeline. In 5.1.2 we see this:

  • The file that also defines pipeline.inputs (the “main” pipeline) keeps its outputs.
  • Any other file that only (or also) defines pipeline.outputs, but is loaded before that main file, has those outputs silently dropped.
  • There is no error / warning message observ. Config test can still pass.
  • Files that only add filters still work even if they load first.
  • It is two (or more) pipeline.outputs blocks, and the early ones never become live plugins if they are parsed before the main pipeline file.

Typical files:

  • extra-output.yml — pipeline.outputs only (second destination: kafka or extra stdout). Name starts with E, so it loads first.
  • filters.yml — pipeline.filters only.
  • main.yml — pipeline.inputs and pipeline.outputs (primary destination). Name starts with M, so it loads after extra.
  • Both extra-output.yml and main.yml define pipeline.outputs. That is two output lists for one process.

Expected: merge appends all outputs (main plus extra), like filters.

Actual: only main.yml outputs run. Extra file outputs never appear in logs or /api/v1/metrics.

Proof it is load order, not the extra YAML: rename extra-output.yml to zz-extra-output.yml so it loads after main.yml. Restart. Both output lists are live. Same content, only the filename (order) changed. Extra outputs show up as later plugin ids (for example kafka.5 while main still has loki.0 or stdout.0), so they were appended after the main list, not a full replace of main.

Extra loads before main → extra outputs are gone.
Extra loads after main → both lists exist.

How to reproduce

fluent-bit.yml
conf/apps/extra-output.yml   # has pipeline.outputs — sorts BEFORE main.yml
conf/apps/main.yml           # has pipeline.inputs AND pipeline.outputs
fluent-bit.yml:

service:
  flush: 1
  log_level: info
  http_server: on
  http_listen: 127.0.0.1
  http_port: 2020
includes:
  - conf/apps/*.yml

conf/apps/extra-output.yml (second file with outputs, no inputs):

pipeline:
  outputs:
    - name: stdout
      match: "demo"
      alias: extra_stdout

conf/apps/main.yml (inputs plus its own outputs):

pipeline:
  inputs:
    - name: dummy
      dummy: '{"msg":"hello"}'
      tag: demo
      rate: 1
  outputs:
    - name: stdout
      match: "demo"
      alias: main_stdout

fluent-bit -c fluent-bit.yml
curl -sS http://127.0.0.1:2020/api/v1/metrics

Bug: metrics show only main_stdout. Dummy prints once. No extra_stdout. No error.

mv conf/apps/extra-output.yml conf/apps/zz-extra-output.yml
# restart fluent-bit

After rename: main_stdout and extra_stdout. Dummy prints twice.

Same with a kafka output in the early file (plugin in the binary, YAML valid): no kafka until that file sorted after main.yml and the process restarted.

Environment

Fluent Bit 5.1.2, Linux, YAML, HTTP API on port 2020.

Docs gap

The Includes page shows a literal list of files, not *.yml, and not what happens if two files both set pipeline.outputs. Pipeline docs say multiple pipeline sections merge into one pipeline. They do not say early output lists can disappear.

Questions

  • Is this a known bug? If yes, please link the issue
  • what should users do today (5.1.x) that fluent bit support ?
  • Can Fluent Bit keep / concatenate pipeline.outputs from every included file, in any order?

Thanks!!

主要言語
C
スター
8.2k
フォーク
2k
平均マージ
3日 20時間
マージ済み PR（30日）
99

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

fluent/fluent-bit のほかの issue

fluent/fluent-bit の issue をすべて見る

似ている issue

C の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。