`component_sent_bytes_total` for the `loki` sink reports compressed bytes, unlike other sinks

Aperta Adatta ai principianti
  • Qualcosa non va?
#26,510 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@ashokram-si ci sta già lavorando.

Dal 27/10/2026.

  • #26886 di @ashokram-si — aperta

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
88/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
observability-sre

Direzione di ricerca

Inizia da src/sinks/util/request_builder.rs e confronta i relativi metadati di encoding con quelli di src/sinks/datadog/logs/sink.rs, quindi segui il valore attraverso src/sinks/util/metadata.rs, lib/vector-common/src/request_metadata.rs e src/sinks/loki/service.rs. Aggiungi o aggiorna test mirati per l'encoding compresso e verifica che component_sent_bytes_total riporti byte non compressi per Loki e per gli altri sink che utilizzano il builder condiviso.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

A note for the community

Vector version: 0.57.0

Summary

The loki sink reports component_sent_bytes_total after compression. Other sinks, such as datadog_logs, report it before compression, which is what the component spec and #20610 say it should be.

The result is that the same logs sent to two sinks show very different sizes. That makes the metric misleading for capacity planning and cost comparisons.

What we see

We send the same log stream to both a datadog_logs sink and a loki sink (snappy compression). We compare component_sent_bytes_total to component_sent_event_bytes_total for each:

Sink sent_bytes ÷ sent_event_bytes
datadog_logs ~1.0
loki ~0.2
  • For datadog_logs, bytes sent roughly equals event bytes, which is what we expect.
  • For loki, bytes sent is about 5× smaller, which fits the compressed payload size rather than the uncompressed one. Loki's own ingest metrics agree with the event-bytes figure.
Why it happens

The loki sink uses the shared default RequestBuilder::encode_events. That function compresses the payload first, then passes the compressed length into the slot meant for the uncompressed size:

That value then flows into the metric:

  1. src/sinks/util/metadata.rs#L93-L99: uncompressed_byte_size becomes request_encoded_size.
  2. lib/vector-common/src/request_metadata.rs#L270-L310: it is stored and returned as-is.
  3. src/sinks/loki/service.rs#L56-L57: bytes_sent() returns request_encoded_size().

By contrast, datadog_logs has its own builder. It measures the size before compressing:

Expected

component_sent_bytes_total should report uncompressed bytes for every sink, including loki. Any other siult encode_events with compression turned on is probably affected too.

Suggested fix

In the default encode_events, record the payload length before compression and pass that as uncompressed_datadog_logs does.

Problem

component_sent_bytes_total for the loki sink reports compressed bytes, unlike other sinks

Configuration
Version

0.57.0

Debug Output
Example Data

No response

Additional Context

No response

References

No response

Lingua principale
Rust
Stelle
22.8k
Fork
2.4k
Merge medio
1g 13h
PR unite (30g)
273

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di vectordotdev/vector

Tutte le issue di vectordotdev/vector

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.