Option to pause inputs instead of dropping chunks when storage.total_limit_size is reached

Aperta
  • Qualcosa non va?
#12,693 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
38/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
c
Ambito
stream-processing

Direzione di ricerca

Start by tracing how storage.total_limit_size currently drops chunks and how the tail input is paused and resumed. Check how chunks left from before a restart are counted. Done means inputs routed to a full output pause without losing chunks, then resume when storage has room.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Is your feature request related to a problem? Please describe.

We use the tail input with filesystem storage and retry_limit: no_limits. Our source files stay on disk with seperate retention, so during an output outage we don't need Fluent Bit to buffer everything. Pausing tail would be enough, since it can pick up later from the DB offset.

We found no way to bound the buffer without losing data:

  • storage.pause_on_chunks_overlimit + storage.max_chunks_up only counts chunks that are up. Chunks waiting for retry, and chunks loaded after a restart, are down, so tail keeps reading. On 5.1.5 with max_chunks_up: 10, the buffer was 34 chunks / 42 MB after a ~2.5 min outage, and 84 chunks / 121 MB after 5 restarts (+10 chunks each time). I understand this may be by design, since max_chunks_up is about memory.
  • storage.total_limit_size bounds the size, but it drops the oldest chunks. tail has already moved past that data, so it's lost for good.

Describe the solution you'd like

Let storage.total_limit_size pause the inputs routed to that output instead of dropping chunks, and resume them once there is room again. It should also count the chunks left over from before a restart. For example:

outputs:
  - name: http
    storage.total_limit_size: 2G
    storage.total_limit_action: pause   # default: drop_oldest

Describe alternatives you've considered

  • Watching the storage dir from outside and stopping Fluent Bit: works, but it's hacky.

Additional context

The use case is robots that write JSONL locally and upload when the network is available. Outages can last hours or days, and the process can restart during one. We'd like to keep disk usage bounded without dropping data, as like fluentbit does for memory.

I'm happy to work on a PR if this direction makes sense.

Lingua principale
C
Stelle
8.2k
Fork
2k
Merge medio
3g 20h
PR unite (30g)
99

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di fluent/fluent-bit

Tutte le issue di fluent/fluent-bit

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.