out_opentelemetry: option to exclude specific body keys from log attributes (avoid duplication with logs_*_message_key)

Aperta
  • Qualcosa non va?
#12,735 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
55/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
c
Ambito
observability

Direzione di ricerca

Start by reading log_record_set_attributes() and the handling of logs_body_key and the logs_*_message_key accessors in the OpenTelemetry output plugin. Check the related tests for body fields converted into log attributes, then determine how excluded keys should be configured and skipped. Done means mapped fields no longer appear twice while other body fields remain attributes; add coverage for that behavior.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem

When logs_body_key_attributes: true is used together with a
logs_*_message_key, a record field that is mapped into a first-class
OTLP field is also emitted as a log attribute, so the same value
appears twice on the wire.

logs_body_key is the only key that gets "consumed":
log_record_set_attributes() removes it via flb_mp_accessor_keys_remove()
before the leftover body fields are converted into attributes. The
logs_trace_id_message_key, logs_severity_text_message_key and
logs_span_id_message_key accessors (and the resource accessor
$resource['attributes']) only read the field — they never remove it —
so it is swept into attributes as well.

Repro
outputs:
  - name: opentelemetry
    match: "*"
    logs_body_key: message
    logs_body_key_attributes: true
    # defaults:
    #   logs_trace_id_message_key: $TraceId
    #   logs_severity_text_message_key: $SeverityText

Given a record {message, Timestamp, TraceId, SeverityText, thread, logger, filename},
the exported LogRecord is:

  • body = message
  • trace_id =
  • severity_text =
  • attributes = [Timestamp, TraceId, SeverityText, thread, logger, filename, ...]

TraceId, SeverityText (and Timestamp) appear both as the first-class
field and as an attribute, inflating backend storage and surfacing as
duplicate columns.

Why logs_body_key_attributes: false is not an acceptable workaround

Setting it to false removes the duplicated attributes, but it also drops
every other field we intentionally want as attributes — e.g. thread,
logger, filename. There is no way to keep a subset:

logs_body_key: message
logs_body_key_attributes: false   # also loses thread/logger/filename

We keep logs_body_key_attributes: true precisely so that those fields
are preserved as attributes, but this is what causes TraceId /
SeverityText to be re-emitted. There is no option to say
"keep attributes, but exclude these keys".

Expected

A way to prevent the mapped keys from being re-emitted as attributes,
mirroring how logs_body_key is removed — while keeping the remaining
body fields as attributes.

Proposal

Add e.g. logs_body_key_attributes_exclude (comma-separated keys) whose
entries are skipped during the leftover-attributes sweep — or let the
logs_*_message_key accessors consume their matched key when
logs_body_key_attributes: true.

Current workaround

Move the keys into metadata.otlp.* via a 5-argument Lua filter;
log_record_set_attributes() skips the otlp metadata key entirely, so
those values are not emitted as attributes. This works, but it requires a
5-arg Lua callback and per-field bookkeeping that the proposed option
would make unnecessary.

Version

Fluent Bit v5.2.0 (master)

Lingua principale
C
Stelle
8.2k
Fork
2k
Merge medio
3g 20h
PR unite (30g)
99

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di fluent/fluent-bit

Tutte le issue di fluent/fluent-bit

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.