out_opentelemetry: option to exclude specific body keys from log attributes (avoid duplication with logs_*_message_key)

Abierto
  • ¿Algo no está bien?
#12,735 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
55/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
c
Área
observability

Línea de trabajo

Start by reading log_record_set_attributes() and the handling of logs_body_key and the logs_*_message_key accessors in the OpenTelemetry output plugin. Check the related tests for body fields converted into log attributes, then determine how excluded keys should be configured and skipped. Done means mapped fields no longer appear twice while other body fields remain attributes; add coverage for that behavior.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

When logs_body_key_attributes: true is used together with a
logs_*_message_key, a record field that is mapped into a first-class
OTLP field is also emitted as a log attribute, so the same value
appears twice on the wire.

logs_body_key is the only key that gets "consumed":
log_record_set_attributes() removes it via flb_mp_accessor_keys_remove()
before the leftover body fields are converted into attributes. The
logs_trace_id_message_key, logs_severity_text_message_key and
logs_span_id_message_key accessors (and the resource accessor
$resource['attributes']) only read the field — they never remove it —
so it is swept into attributes as well.

Repro
outputs:
  - name: opentelemetry
    match: "*"
    logs_body_key: message
    logs_body_key_attributes: true
    # defaults:
    #   logs_trace_id_message_key: $TraceId
    #   logs_severity_text_message_key: $SeverityText

Given a record {message, Timestamp, TraceId, SeverityText, thread, logger, filename},
the exported LogRecord is:

  • body = message
  • trace_id =
  • severity_text =
  • attributes = [Timestamp, TraceId, SeverityText, thread, logger, filename, ...]

TraceId, SeverityText (and Timestamp) appear both as the first-class
field and as an attribute, inflating backend storage and surfacing as
duplicate columns.

Why logs_body_key_attributes: false is not an acceptable workaround

Setting it to false removes the duplicated attributes, but it also drops
every other field we intentionally want as attributes — e.g. thread,
logger, filename. There is no way to keep a subset:

logs_body_key: message
logs_body_key_attributes: false   # also loses thread/logger/filename

We keep logs_body_key_attributes: true precisely so that those fields
are preserved as attributes, but this is what causes TraceId /
SeverityText to be re-emitted. There is no option to say
"keep attributes, but exclude these keys".

Expected

A way to prevent the mapped keys from being re-emitted as attributes,
mirroring how logs_body_key is removed — while keeping the remaining
body fields as attributes.

Proposal

Add e.g. logs_body_key_attributes_exclude (comma-separated keys) whose
entries are skipped during the leftover-attributes sweep — or let the
logs_*_message_key accessors consume their matched key when
logs_body_key_attributes: true.

Current workaround

Move the keys into metadata.otlp.* via a 5-argument Lua filter;
log_record_set_attributes() skips the otlp metadata key entirely, so
those values are not emitted as attributes. This works, but it requires a
5-arg Lua callback and per-field bookkeeping that the proposed option
would make unnecessary.

Version

Fluent Bit v5.2.0 (master)

Lenguaje dominante
C
Estrellas
8.2k
Forks
2k
Merge medio
3 d 20 h
PR fusionados (30 d)
99

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de fluent/fluent-bit

Todos los issues de fluent/fluent-bit

Issues similares

Más issues de C

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.