out_forward: unacknowledged chunks are resent forever with heartbeat_type none, ignoring retry limits and secondary

Abierto
  • ¿Algo no está bien?
#5,619 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@SahinurDEV ya está trabajando en esto.

Desde el 10/11/2026.

  • #5627 de @SahinurDEV — abierto

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
74/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
ruby
Área
backend, distributed-systems

Línea de trabajo

Start with lib/fluent/plugin/out_forward.rb around ack_check and rollback_write, then read lib/fluent/plugin/out_forward/ack_handler.rb around collect_response. Trace how ACK timeouts and closed connections affect retry state when heartbeat_type none is used. Add coverage for the reproduced configuration and verify failed acknowledgments honor the retry limits and secondary behavior.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug
Describe the bug

With require_ack_response true and heartbeat_type none, a chunk that the destination never acknowledges is resent forever. retry_timeout, retry_max_times and <secondary> never apply.

When the ACK times out or the destination closes the connection, ack_check calls rollback_write(chunk_id, update_retry: false) (lib/fluent/plugin/out_forward.rb:547). This puts the chunk back to the queue without starting a retry state, so the next flush sends it again at once. With the default heartbeat_type transport, node&.disable! on line 546 makes the next write raise NoNodesAvailable, and that starts the retry state. NoneHeartbeatNode#available? always returns true, so heartbeat_type none never reaches that path.

A destination that rejects a chunk is one way to hit this. in_forward drops a chunk larger than chunk_size_limit and sends no ACK.

To Reproduce
  1. Save the configuration below as fluentd.conf.
  2. Run fluentd -c fluentd.conf for about 40 seconds and stop it.
  3. Count the log lines and the files in the secondary directory.
$ grep -c "larger than 'chunk_size_limit', dropped" fluentd.log
66
$ grep -c "destination node closed the connection" fluentd.log
66
$ find /tmp/fluentd-secondary -type f | wc -l
0

sample creates one chunk per second, but the destination dropped 66 chunks in about 37 seconds. The same chunks were sent again and again. Nothing was written to the secondary although retry_timeout is 10 seconds. The only failed to flush the buffer line came at shutdown, after the destination had stopped (Errno::ECONNREFUSED).

Without deny_keepalive true, the destination keeps the connection open and the ACK times out instead. Adding ack_response_timeout 3s to <match> makes this quick. The result is the same: 45 no response from node warnings, no retry and no secondary file in about 37 seconds.

With the default heartbeat_type transport (remove the heartbeat_type none line), the retry state starts and the chunks go to the secondary (retry succeeded by secondary).

Expected behavior

A chunk that is never acknowledged follows the retry settings, as it does with heartbeat_type transport or when the write raises an error. It should be retried with backoff, go to <secondary> after retry_secondary_threshold, and stop after retry_timeout or retry_max_times.

Your Environment
- Fluentd version: 1.20.1 (master at 73976761)
- Package version: none (git checkout)
- Operating system: CachyOS
- Kernel version: 7.3.1-1-cachyos
- Ruby: 4.0.9
Your Configuration
<source>
  @type sample
  tag test
  rate 1
  size 1
</source>

<match test>
  @type forward
  require_ack_response true
  heartbeat_type none
  <server>
    host 127.0.0.1
    port 24411
  </server>
  <buffer>
    flush_interval 1s
    retry_timeout 10s
    retry_max_times 2
  </buffer>
  <secondary>
    @type secondary_file
    directory /tmp/fluentd-secondary
  </secondary>
</match>

# The destination. It drops every chunk and closes the connection.
<source>
  @type forward
  @label @IN
  bind 127.0.0.1
  port 24411
  chunk_size_limit 10
  deny_keepalive true
</source>

<label @IN>
  <match **>
    @type null
  </match>
</label>
Your Error Log
2026-11-09 14:23:36 +0900 [warn]: #0 Input chunk size is larger than 'chunk_size_limit', dropped: tag="test" host="127.0.0.1" limit=10 size=120
2026-11-09 14:23:36 +0900 [warn]: #0 destination node closed the connection. regard it as unavailable. host="127.0.0.1" port=24411
2026-11-09 14:23:37 +0900 [warn]: #0 Input chunk size is larger than 'chunk_size_limit', dropped: tag="test" host="127.0.0.1" limit=10 size=120
2026-11-09 14:23:37 +0900 [warn]: #0 destination node closed the connection. regard it as unavailable. host="127.0.0.1" port=24411
(the same pair repeats until shutdown)
Additional context

The ACK timeout path in AckHandler#collect_response (lib/fluent/plugin/out_forward/ack_handler.rb:53) also ends in the same rollback_write(..., update_retry: false). #5607 sends a connection closed with FIN (where recv returns nil) to this path too. Today a FIN close raises NoMethodError and the chunk waits for the delayed commit timeout, which does update the retry state.

Lenguaje dominante
Ruby
Estrellas
13.6k
Forks
1.4k
Merge medio
16 h 41 min
PR fusionados (30 d)
70

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de fluent/fluentd

Todos los issues de fluent/fluentd

Issues similares

Más issues de Ruby

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.