Option to pause inputs instead of dropping chunks when storage.total_limit_size is reached

オープン
  • 何か問題がありますか？
#12,693 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
38/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
c
領域
stream-processing

調査の方向性

Start by tracing how storage.total_limit_size currently drops chunks and how the tail input is paused and resumed. Check how chunks left from before a restart are counted. Done means inputs routed to a full output pause without losing chunks, then resume when storage has room.

索引モデルが issue の本文から書いたものです。

説明

Is your feature request related to a problem? Please describe.

We use the tail input with filesystem storage and retry_limit: no_limits. Our source files stay on disk with seperate retention, so during an output outage we don't need Fluent Bit to buffer everything. Pausing tail would be enough, since it can pick up later from the DB offset.

We found no way to bound the buffer without losing data:

  • storage.pause_on_chunks_overlimit + storage.max_chunks_up only counts chunks that are up. Chunks waiting for retry, and chunks loaded after a restart, are down, so tail keeps reading. On 5.1.5 with max_chunks_up: 10, the buffer was 34 chunks / 42 MB after a ~2.5 min outage, and 84 chunks / 121 MB after 5 restarts (+10 chunks each time). I understand this may be by design, since max_chunks_up is about memory.
  • storage.total_limit_size bounds the size, but it drops the oldest chunks. tail has already moved past that data, so it's lost for good.

Describe the solution you'd like

Let storage.total_limit_size pause the inputs routed to that output instead of dropping chunks, and resume them once there is room again. It should also count the chunks left over from before a restart. For example:

outputs:
  - name: http
    storage.total_limit_size: 2G
    storage.total_limit_action: pause   # default: drop_oldest

Describe alternatives you've considered

  • Watching the storage dir from outside and stopping Fluent Bit: works, but it's hacky.

Additional context

The use case is robots that write JSONL locally and upload when the network is available. Outages can last hours or days, and the process can restart during one. We'd like to keep disk usage bounded without dropping data, as like fluentbit does for memory.

I'm happy to work on a PR if this direction makes sense.

主要言語
C
スター
8.2k
フォーク
2k
平均マージ
3日 20時間
マージ済み PR（30日）
99

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

fluent/fluent-bit のほかの issue

fluent/fluent-bit の issue をすべて見る

似ている issue

C の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。