out_forward: unacknowledged chunks are resent forever with heartbeat_type none, ignoring retry limits and secondary

オープン
  • 何か問題がありますか？
#5,619 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@SahinurDEV がすでに取り組んでいます。

2026年11月10日 から。

  • #5627 @SahinurDEV による — オープン

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
74/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
ruby
領域
backend, distributed-systems

調査の方向性

Start with lib/fluent/plugin/out_forward.rb around ack_check and rollback_write, then read lib/fluent/plugin/out_forward/ack_handler.rb around collect_response. Trace how ACK timeouts and closed connections affect retry state when heartbeat_type none is used. Add coverage for the reproduced configuration and verify failed acknowledgments honor the retry limits and secondary behavior.

索引モデルが issue の本文から書いたものです。

説明

bug
Describe the bug

With require_ack_response true and heartbeat_type none, a chunk that the destination never acknowledges is resent forever. retry_timeout, retry_max_times and <secondary> never apply.

When the ACK times out or the destination closes the connection, ack_check calls rollback_write(chunk_id, update_retry: false) (lib/fluent/plugin/out_forward.rb:547). This puts the chunk back to the queue without starting a retry state, so the next flush sends it again at once. With the default heartbeat_type transport, node&.disable! on line 546 makes the next write raise NoNodesAvailable, and that starts the retry state. NoneHeartbeatNode#available? always returns true, so heartbeat_type none never reaches that path.

A destination that rejects a chunk is one way to hit this. in_forward drops a chunk larger than chunk_size_limit and sends no ACK.

To Reproduce
  1. Save the configuration below as fluentd.conf.
  2. Run fluentd -c fluentd.conf for about 40 seconds and stop it.
  3. Count the log lines and the files in the secondary directory.
$ grep -c "larger than 'chunk_size_limit', dropped" fluentd.log
66
$ grep -c "destination node closed the connection" fluentd.log
66
$ find /tmp/fluentd-secondary -type f | wc -l
0

sample creates one chunk per second, but the destination dropped 66 chunks in about 37 seconds. The same chunks were sent again and again. Nothing was written to the secondary although retry_timeout is 10 seconds. The only failed to flush the buffer line came at shutdown, after the destination had stopped (Errno::ECONNREFUSED).

Without deny_keepalive true, the destination keeps the connection open and the ACK times out instead. Adding ack_response_timeout 3s to <match> makes this quick. The result is the same: 45 no response from node warnings, no retry and no secondary file in about 37 seconds.

With the default heartbeat_type transport (remove the heartbeat_type none line), the retry state starts and the chunks go to the secondary (retry succeeded by secondary).

Expected behavior

A chunk that is never acknowledged follows the retry settings, as it does with heartbeat_type transport or when the write raises an error. It should be retried with backoff, go to <secondary> after retry_secondary_threshold, and stop after retry_timeout or retry_max_times.

Your Environment
- Fluentd version: 1.20.1 (master at 73976761)
- Package version: none (git checkout)
- Operating system: CachyOS
- Kernel version: 7.3.1-1-cachyos
- Ruby: 4.0.9
Your Configuration
<source>
  @type sample
  tag test
  rate 1
  size 1
</source>

<match test>
  @type forward
  require_ack_response true
  heartbeat_type none
  <server>
    host 127.0.0.1
    port 24411
  </server>
  <buffer>
    flush_interval 1s
    retry_timeout 10s
    retry_max_times 2
  </buffer>
  <secondary>
    @type secondary_file
    directory /tmp/fluentd-secondary
  </secondary>
</match>

# The destination. It drops every chunk and closes the connection.
<source>
  @type forward
  @label @IN
  bind 127.0.0.1
  port 24411
  chunk_size_limit 10
  deny_keepalive true
</source>

<label @IN>
  <match **>
    @type null
  </match>
</label>
Your Error Log
2026-11-09 14:23:36 +0900 [warn]: #0 Input chunk size is larger than 'chunk_size_limit', dropped: tag="test" host="127.0.0.1" limit=10 size=120
2026-11-09 14:23:36 +0900 [warn]: #0 destination node closed the connection. regard it as unavailable. host="127.0.0.1" port=24411
2026-11-09 14:23:37 +0900 [warn]: #0 Input chunk size is larger than 'chunk_size_limit', dropped: tag="test" host="127.0.0.1" limit=10 size=120
2026-11-09 14:23:37 +0900 [warn]: #0 destination node closed the connection. regard it as unavailable. host="127.0.0.1" port=24411
(the same pair repeats until shutdown)
Additional context

The ACK timeout path in AckHandler#collect_response (lib/fluent/plugin/out_forward/ack_handler.rb:53) also ends in the same rollback_write(..., update_retry: false). #5607 sends a connection closed with FIN (where recv returns nil) to this path too. Today a FIN close raises NoMethodError and the chunk waits for the delayed commit timeout, which does update the retry state.

主要言語
Ruby
スター
13.6k
フォーク
1.4k
平均マージ
16時間 41分
マージ済み PR（30日）
70

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

fluent/fluentd のほかの issue

fluent/fluentd の issue をすべて見る

似ている issue

Ruby の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。