Option to pause inputs instead of dropping chunks when storage.total_limit_size is reached

Abierto
  • ¿Algo no está bien?
#12,693 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
38/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
c
Área
stream-processing

Línea de trabajo

Start by tracing how storage.total_limit_size currently drops chunks and how the tail input is paused and resumed. Check how chunks left from before a restart are counted. Done means inputs routed to a full output pause without losing chunks, then resume when storage has room.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Is your feature request related to a problem? Please describe.

We use the tail input with filesystem storage and retry_limit: no_limits. Our source files stay on disk with seperate retention, so during an output outage we don't need Fluent Bit to buffer everything. Pausing tail would be enough, since it can pick up later from the DB offset.

We found no way to bound the buffer without losing data:

  • storage.pause_on_chunks_overlimit + storage.max_chunks_up only counts chunks that are up. Chunks waiting for retry, and chunks loaded after a restart, are down, so tail keeps reading. On 5.1.5 with max_chunks_up: 10, the buffer was 34 chunks / 42 MB after a ~2.5 min outage, and 84 chunks / 121 MB after 5 restarts (+10 chunks each time). I understand this may be by design, since max_chunks_up is about memory.
  • storage.total_limit_size bounds the size, but it drops the oldest chunks. tail has already moved past that data, so it's lost for good.

Describe the solution you'd like

Let storage.total_limit_size pause the inputs routed to that output instead of dropping chunks, and resume them once there is room again. It should also count the chunks left over from before a restart. For example:

outputs:
  - name: http
    storage.total_limit_size: 2G
    storage.total_limit_action: pause   # default: drop_oldest

Describe alternatives you've considered

  • Watching the storage dir from outside and stopping Fluent Bit: works, but it's hacky.

Additional context

The use case is robots that write JSONL locally and upload when the network is available. Outages can last hours or days, and the process can restart during one. We'd like to keep disk usage bounded without dropping data, as like fluentbit does for memory.

I'm happy to work on a PR if this direction makes sense.

Lenguaje dominante
C
Estrellas
8.2k
Forks
2k
Merge medio
3 d 20 h
PR fusionados (30 d)
99

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de fluent/fluent-bit

Todos los issues de fluent/fluent-bit

Issues similares

Más issues de C

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.