buf_file: queued file chunks keep FDs open until shutdown; resume opens all chunks — EMFILE death spiral on slow output

Aperta
  • Qualcosa non va?
#5,494 2 commenti 1 reazione 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
ruby
Ambito
backend, performance

Direzione di ricerca

Inizia leggendo lib/fluent/plugin/buffer.rb, lib/fluent/plugin/buffer/file_chunk.rb e lib/fluent/plugin/buf_file.rb, concentrandoti sul comportamento di enqueue, resume e close. Riproduci lo scenario EMFILE con un grande backlog di file-buffer accodati; il lavoro è completato quando esiste una modifica concordata del ciclo di vita che consenta a Fluentd di recuperare e smaltire il backlog senza richiedere uno o due FDs aperti per chunk.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug enhancement
Describe the bug

When using @type file buffers with a slow or failing output (S3, Kafka, forward, etc.), Fluentd accumulates many buffer chunk files. Each chunk keeps one or two file descriptors open while it sits in stage or queue, and buf_file#resume re-opens every persisted chunk on startup. Once open FD count approaches the process limit (often 65536 on Linux), Fluentd enters a death spiral:

  1. Output flush fails with Errno::EMFILE (cannot open TCP socket / DNS / temp files)
  2. Queued chunks are not purged and their FDs are not released
  3. New chunks cannot be created (BufferOverflowError)
  4. Even after downstream recovers, Fluentd may remain stuck because resume/flush paths need more FDs than available

This is related to long-standing reports (#1612, #3040, #4012, #3993), but those issues focus on symptoms/workarounds (raise ulimit, tune buffer size). This report points at a specific FD lifecycle design in current buf_file / FileChunk that makes recovery impossible at scale without deleting buffers or raising limits to match chunk count.

Root cause in code (v1.x)

1. enqueue_chunk closes only empty chunks — non-empty queued chunks stay open

# lib/fluent/plugin/buffer.rb
if chunk.empty?
  chunk.close
else
  @queue << chunk
  chunk.enqueued!  # renames files but keeps @chunk and @meta open
end

2. FileChunk#enqueued! keeps both @chunk and @meta handles open after rename

After staging, each chunk uses 2 FDs (@chunk + @meta). After enqueue, both handles remain open through flush (see file_rename callbacks in lib/fluent/plugin/buffer/file_chunk.rb). The code even notes:

"Too many open files" should be fixed by proper buffer configuration and system setting.

3. buf_file#resume opens every buffer file during startup

# lib/fluent/plugin/buf_file.rb
Dir.glob(escaped_patterns(patterns)) do |path|
  chunk = Fluent::Plugin::Buffer::FileChunk.new(m, path, mode, ...)
  queue << chunk  # File.open in load_existing_enqueued_chunk / load_existing_staged_chunk
end

With tens of thousands of backlog files, resume alone can exhaust FDs before any flush thread runs (#3040).

4. FDs are only released on buffer shutdown

Buffer#close closes dequeued, queued, and staged chunks — not when a chunk moves from stage → queue.

PR #1468 (v0.14.12) handles EMFILE on chunk create by raising BufferOverflowError, but does not close existing queued chunk FDs or enable recovery.

To Reproduce
  1. Configure a file buffer with relatively small timekey and large chunk_limit_size, e.g. S3 output:
<match **>
  @type s3
  # ... aws / bucket config ...
  <buffer tag,time>
    @type file
    path /var/log/fluent/buffer
    timekey 5
    chunk_limit_size 100MB
    flush_mode interval
    flush_interval 5
    flush_thread_count 15
    # note: no total_limit_size — backlog can grow by chunk count
  </buffer>
</match>

  1. Block or fail the output path (wrong credentials, network partition, unreachable S3 endpoint) for long enough to accumulate ~30k+ chunks (≈ 60k+ FDs with chunk+meta open, or ~32k chunks at 65536 limit).

  2. Observe logs:

failed to flush the buffer. retry_times=... error="Too many open files" ...
Failed to open TCP connection to ...s3.amazonaws.com:443
can't create buffer metadata for ... error = Too many open files @ rb_sysopen

  1. Restore downstream — Fluentd often does not recover without manual intervention (delete buffer dir or raise LimitNOFILE above chunk count).

  2. Restart Fluentd with large backlog — restoring buffer file storm in logs; process hits EMFILE during resume before flushing (#3040).

Expected behavior

File buffer chunks should not require a permanently open FD for every queued chunk. Suggested approaches (any of these would help):

  1. Close on enqueue: after enqueued!, call chunk.close (or equivalent) so queued chunks hold 0 FDs until dequeued for flush.
  2. Lazy open on flush: FileChunk#open / read reopen paths on demand; keep metadata in memory or mmap only .meta briefly.
  3. Lazy resume: do not File.open all chunks in Dir.glob; index paths from filesystem and open in batches bounded by (flush_thread_count * 2) or configurable resume_open_limit.
  4. Document FD estimation: queue_limit_length, total_limit_size, and chunk count vs LimitNOFILE (as suggested in #3040).

At minimum, Fluentd should remain able to drain an existing backlog after downstream recovery without requiring FD count ≥ chunk count.

Actual behavior
  • ~1–2 FDs per staged/queued chunk for the lifetime of the chunk in memory
  • Resume opens all chunks eagerly
  • EMFILE on output flush → permanent stall until buffer deletion or ulimit >> chunk count
  • BufferOverflowError on create stops ingestion but does not free queued FDs
Your Environment
- Fluentd: 1.16+ / td-agent 4.x (behavior present since v0.14 file buffer redesign)
- OS: Linux (Kubernetes), `LimitNOFILE` commonly 65536
- Buffer: `@type file` with timekey slicing, S3 or other slow output
- Observed at production scale: ~65k FD limit reached with ~32k queued chunks (chunk + meta open)
Your Error Log
failed to flush the buffer. retry_times=0 next_retry_time=... error="Too many open files"
  error_class=Errno::EMFILE
Failed to open TCP connection to <bucket>.s3.amazonaws.com:443

emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError
error="can't create buffer metadata for /var/log/fluent/buffer/buffer.*.log.
Stop creating buffer files: error = Too many open files @ rb_sysopen - ...log.meta"
Additional context
  • Community has reported this pattern since 2017 (#1612: 131034 files, 65516 .log + 65515 .meta, exactly at 65536 ulimit).
  • #3040 (still open) requests batch open/close on resume — no upstream fix yet.
  • Workarounds today: raise ulimit, delete buffer files (data loss), or cap backlog via total_limit_size / overflow_action — none fix FD retention on queued chunks.
  • Willing to contribute a PR for close-on-enqueue + lazy reopen if maintainers agree on direction.
Lingua principale
Ruby
Stelle
13.6k
Fork
1.4k
Merge medio
1g 4h
PR unite (30g)
56

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di fluent/fluentd

Tutte le issue di fluent/fluentd

Issue simili

Altre issue su Ruby

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.