in_tail: pending multiline record silently dropped when flush_timeout fires while the input is paused

Abierto
  • ¿Algo no está bien?
#12,710 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
1-2 días
Aptitud para principiantes
38/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
c
Área
stream-processing

Línea de trabajo

Start in src/multiline/flb_ml.c at cb_ml_flush_timer and flb_ml_flush_stream_group, where the flush result is ignored and the group buffer is cleared. Then read ml_stream_buffer_flush in plugins/in_tail/tail_file.c, which drops the encoder buffer after flb_input_log_append fails, and the rejection in input_chunk_append_raw in src/flb_input_chunk.c. Done means a pending record survives a pause: it is either delivered after resume, or the drop is counted and logged, with a regression test for the paused-input case.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

status: waiting-for-triage

Bug Report

Describe the bug

When in_tail uses multiline.parser and the input is paused (either mem_buf_limit overlimit, or storage.type: filesystem with storage.pause_on_chunks_overlimit: on), the multiline flush timer still fires. The pending (last) multiline record is flushed into flb_input_log_append(), which rejects it because the input is paused ([input chunk] tail.0 is paused, cannot append records). The return value is ignored and both the multiline group buffer and in_tail's encoder buffer are reset, so the record is permanently lost. The file offset has already advanced past those lines, so it is never re-read, and no metric (dropped_records etc.) is incremented.

This is distinct from #12643 (shutdown) — it happens during normal operation whenever the output is slow/unavailable long enough for the input to stay paused for longer than flush_timeout.

To Reproduce

  • Example log message (three multiline records written in a single append):
13:10:01:000|7001-ERROR recA start
  recA cont
13:10:02:000|7002-ERROR recB start
  recB cont
13:10:03:000|7003-ERROR recC start
  recC cont
  • Multiline parser:
multiline_parsers:
  - name: multiline_MOCK_COMP
    type: regex
    flush_timeout: 1000
    rules:
      - state: start_state
        regex: /(^[012][0-9]:[0-5][0-9]:[0-6][0-9]:[0-9][0-9][0-9]([0-9][0-9][0-9])?\|[0-9][0-9][0-9][0-9]-)/
        next_state: cont
      - state: cont
        regex: /^(?![012][0-9]:[0-5][0-9]:[0-6][0-9]:[0-9][0-9][0-9]([0-9][0-9][0-9])?\|[0-9][0-9][0-9][0-9]-)/
        next_state: cont
  • Pipeline:
service:
  flush: 1
  log_level: debug

includes:
  - parsers.yaml

pipeline:
  inputs:
    - name: tail
      path: /logs/*.log
      tag: mock.comp
      read_from_head: true
      multiline.parser: multiline_MOCK_COMP
      mem_buf_limit: 70        # small to force a pause quickly
  outputs:
    - name: forward
      match: '*'
      host: fluentd
      port: 24224
      retry_limit: no_limits

  • Steps to reproduce the problem:

    1. Start Fluent Bit and a Fluentd receiver (<source> @type forward → <match **> @type stdout).
    2. Stop the receiver so chunks cannot be delivered.
    3. Append the six lines above to the tailed file in a single write.
    4. recA and recB are completed and appended in one batch; the input goes overlimit and pauses. recC remains pending in the multiline parser.
    5. ~1s later the flush timer fires and recC is rejected: 
      [ warn] [input] tail.0 paused (mem buf overlimit - event of size 146B exceeded limit 70 to 146B)
[debug] [input chunk] tail.0 is paused, cannot append records
    6. Start the receiver again. The retried chunk delivers recA and recB; recC never arrives.

    The same result occurs with flush_timeout: 10000 (any timeout shorter than the pause), and with filesystem storage when storage.pause_on_chunks_overlimit: on (log shows paused (storage buf overlimit 1/1) followed by cannot append records, recC lost). With filesystem storage and storage.pause_on_chunks_overlimit: off the input never pauses, so the issue does not occur.

Expected behavior

A pending multiline record flushed by the timer must not be discarded because the input is paused. Either the flush should be deferred while the input is paused (keep the record in the multiline group buffer until the input resumes), or the record should be retained and retried when flb_input_log_append() fails. At minimum, the drop should be counted in a metric and logged above debug level.

Code analysis (master)

  1. The flush timer is a scheduler timer, independent of input pause/resume — src/multiline/flb_ml.c flb_ml_auto_flush_init() → flb_sched_timer_cb_create(..., cb_ml_flush_timer, ...). cb_ml_flush_timer() calls flb_ml_flush_pending(ml, now, FLB_TRUE) regardless of input state.
  2. flb_ml_flush_stream_group() calls mst->cb_flush(...) ignoring its return, then unconditionally clears the group: flb_sds_len_set(group->buf, 0);.
  3. plugins/in_tail/tail_file.c ml_flush_callback() → on forced_flush calls ml_stream_buffer_flush(), which calls flb_input_log_append(...) ignoring the return and then flb_log_event_encoder_reset(file->ml_log_event_encoder);.
  4. src/flb_input_chunk.c input_chunk_append_raw(): 
    if (flb_input_paused(in) == FLB_TRUE && (...not shutting down...)) {
    flb_debug("[input chunk] %s is paused, cannot append records", ...);
    return -1;
}
  5. flb_input_buf_paused() (include/fluent-bit/flb_input.h) is true for both mem_buf_status and storage_buf_status paused, so both pause modes are affected.

Possible fixes: skip cb_ml_flush_timer flushing for streams whose input is paused (re-arm on resume), or have ml_stream_buffer_flush() keep the encoder buffer when flb_input_log_append() returns -1 and retry it on the next read/flush.

Your Environment

  • Version used: Fluent Bit 5.1.5 (Community, docker image fluent/fluent-bit:latest); code analysis against current master
  • Configuration: see above
  • Environment name and version: Docker Desktop (Docker 29.9.2, Compose v5.6.1)
  • Server type and version: Fluentd v1.17.1 (fluent/fluentd:v1.17-1) as forward receiver
  • Operating System and version: Linux containers on Windows 11
  • Filters and plugins: in_tail (multiline.parser, regex type), out_forward
  • Bare Metal Setup

Additional context

The last multiline record in progress (often the tail of a stack trace / error event) is lost whenever the downstream is slow or unavailable long enough to keep the input paused for longer than flush_timeout. This makes mem_buf_limit unsafe to combine with in_tail multiline in at-least-once pipelines; the only workaround is filesystem storage with storage.pause_on_chunks_overlimit: off.

Lenguaje dominante
C
Estrellas
8.2k
Forks
2k
Merge medio
3 d 20 h
PR fusionados (30 d)
99

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de fluent/fluent-bit

Todos los issues de fluent/fluent-bit

Issues similares

Más issues de C

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.