buf_file: queued file chunks keep FDs open until shutdown; resume opens all chunks — EMFILE death spiral on slow output

首先阅读 lib/fluent/plugin/buffer.rb、lib/fluent/plugin/buffer/file_chunk.rb 和 lib/fluent/plugin/buf_file.rb，重点关注 enqueue、resume 和 close 的行为。使用大量排队的 file-buffer backlog 重现 EMFILE 场景；完成条件是确定一项已达成共识的生命周期变更，使 Fluentd 无需为每个 chunk 保持一个或两个打开的 FD 即可恢复并排空 backlog。

Describe the bug

When using @type file buffers with a slow or failing output (S3, Kafka, forward, etc.), Fluentd accumulates many buffer chunk files. Each chunk keeps one or two file descriptors open while it sits in stage or queue, and buf_file#resume re-opens every persisted chunk on startup. Once open FD count approaches the process limit (often 65536 on Linux), Fluentd enters a death spiral:

1. Output flush fails with Errno::EMFILE (cannot open TCP socket / DNS / temp files)
2. Queued chunks are not purged and their FDs are not released
3. New chunks cannot be created (BufferOverflowError)
4. Even after downstream recovers, Fluentd may remain stuck because resume/flush paths need more FDs than available

This is related to long-standing reports (#1612, #3040, #4012, #3993), but those issues focus on symptoms/workarounds (raise ulimit, tune buffer size). This report points at a specific FD lifecycle design in current buf_file / FileChunk that makes recovery impossible at scale without deleting buffers or raising limits to match chunk count.

Root cause in code (v1.x)

1. enqueue_chunk closes only empty chunks — non-empty queued chunks stay open

# lib/fluent/plugin/buffer.rb
if chunk.empty?
  chunk.close
else
  @queue << chunk
  chunk.enqueued!  # renames files but keeps @chunk and @meta open
end

2. FileChunk#enqueued! keeps both @chunk and @meta handles open after rename

After staging, each chunk uses 2 FDs (@chunk + @meta). After enqueue, both handles remain open through flush (see file_rename callbacks in lib/fluent/plugin/buffer/file_chunk.rb). The code even notes:

"Too many open files" should be fixed by proper buffer configuration and system setting.

3. buf_file#resume opens every buffer file during startup

# lib/fluent/plugin/buf_file.rb
Dir.glob(escaped_patterns(patterns)) do |path|
  chunk = Fluent::Plugin::Buffer::FileChunk.new(m, path, mode, ...)
  queue << chunk  # File.open in load_existing_enqueued_chunk / load_existing_staged_chunk
end

With tens of thousands of backlog files, resume alone can exhaust FDs before any flush thread runs (#3040).

4. FDs are only released on buffer shutdown

Buffer#close closes dequeued, queued, and staged chunks — not when a chunk moves from stage → queue.

PR #1468 (v0.14.12) handles EMFILE on chunk create by raising BufferOverflowError, but does not close existing queued chunk FDs or enable recovery.

To Reproduce

1. Configure a file buffer with relatively small timekey and large chunk_limit_size, e.g. S3 output:

<match **>
  @type s3
  # ... aws / bucket config ...
  <buffer tag,time>
    @type file
    path /var/log/fluent/buffer
    timekey 5
    chunk_limit_size 100MB
    flush_mode interval
    flush_interval 5
    flush_thread_count 15
    # note: no total_limit_size — backlog can grow by chunk count
  </buffer>
</match>

2. Block or fail the output path (wrong credentials, network partition, unreachable S3 endpoint) for long enough to accumulate ~30k+ chunks (≈ 60k+ FDs with chunk+meta open, or ~32k chunks at 65536 limit).

3. Observe logs:

failed to flush the buffer. retry_times=... error="Too many open files" ...
Failed to open TCP connection to ...s3.amazonaws.com:443
can't create buffer metadata for ... error = Too many open files @ rb_sysopen

4. Restore downstream — Fluentd often does not recover without manual intervention (delete buffer dir or raise LimitNOFILE above chunk count).

5. Restart Fluentd with large backlog — restoring buffer file storm in logs; process hits EMFILE during resume before flushing (#3040).

Expected behavior

File buffer chunks should not require a permanently open FD for every queued chunk. Suggested approaches (any of these would help):

1. Close on enqueue: after enqueued!, call chunk.close (or equivalent) so queued chunks hold 0 FDs until dequeued for flush.
2. Lazy open on flush: FileChunk#open / read reopen paths on demand; keep metadata in memory or mmap only .meta briefly.
3. Lazy resume: do not File.open all chunks in Dir.glob; index paths from filesystem and open in batches bounded by (flush_thread_count * 2) or configurable resume_open_limit.
4. Document FD estimation: queue_limit_length, total_limit_size, and chunk count vs LimitNOFILE (as suggested in #3040).

At minimum, Fluentd should remain able to drain an existing backlog after downstream recovery without requiring FD count ≥ chunk count.

Actual behavior

• ~1–2 FDs per staged/queued chunk for the lifetime of the chunk in memory
• Resume opens all chunks eagerly
• EMFILE on output flush → permanent stall until buffer deletion or ulimit >> chunk count
• BufferOverflowError on create stops ingestion but does not free queued FDs

Your Environment

- Fluentd: 1.16+ / td-agent 4.x (behavior present since v0.14 file buffer redesign)
- OS: Linux (Kubernetes), `LimitNOFILE` commonly 65536
- Buffer: `@type file` with timekey slicing, S3 or other slow output
- Observed at production scale: ~65k FD limit reached with ~32k queued chunks (chunk + meta open)

Your Error Log

failed to flush the buffer. retry_times=0 next_retry_time=... error="Too many open files"
  error_class=Errno::EMFILE
Failed to open TCP connection to <bucket>.s3.amazonaws.com:443

emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError
error="can't create buffer metadata for /var/log/fluent/buffer/buffer.*.log.
Stop creating buffer files: error = Too many open files @ rb_sysopen - ...log.meta"

Additional context

• Community has reported this pattern since 2017 (#1612: 131034 files, 65516 .log + 65515 .meta, exactly at 65536 ulimit).
• #3040 (still open) requests batch open/close on resume — no upstream fix yet.
• Workarounds today: raise ulimit, delete buffer files (data loss), or cap backlog via total_limit_size / overflow_action — none fix FD retention on queued chunks.
• Willing to contribute a PR for close-on-enqueue + lazy reopen if maintainers agree on direction.