vectordotdev/vector

Allow specifying the object key in the `aws_s3` sink

Open

Aperta il 30 mar 2021

Vedi su GitHub
 (0 commenti) (0 reazioni) (0 assegnatari)Rust (21.837 star) (2126 fork)batch import
good first issuesink: aws_s3type: enhancement

Descrizione

Requested by user migrating from fluentd.

Currently, we expose some configuration knobs for the object prefix + key for the aws_s3 sink:

  filename_append_uuid = true # optional, default
  filename_extension = "log" # optional, default
  filename_time_format = "%s" # optional, default
  key_prefix = "date=%F/" # optional, default

However there is no way to directly set the key. Ideally this could be templated from the event (like key = "{{ host }}"). This would affect internal partitioning of events across S3 requests. It is likely we'll still want to support formatting the timestamp as part of that, as well.

Ref: fluentd's s3 output: https://docs.fluentd.org/output/s3#s-3-_object_key_format

Guida contributor

Tech stack
rustaws
Dominio
backendclouddata
Tipo issue
feature
DifficoltàQuanto dovrebbe essere impegnativa per un nuovo contributor.
3
Tempo stimatoTempo stimato per completare e verificare uno scope piccolo.
half day
Stato attivitàQuanto è probabile che la issue sia ancora attiva e reviewable.
stale
ChiarezzaQuanto chiaramente la issue descrive problema, scope e risultato atteso.
clear
Prerequisiti
Rust programmingAWS S3 basicsVector sink configuration
Adatta ai principiantiQuanto la issue è adatta a contributor alla prima esperienza.
50
Direzione di ricerca
The issue requests adding a configurable key template for AWS S3 sink. Start by examining the existing sink configuration in `src/sinks/aws s3/config.rs` and the key formatting logic in `src/sinks/aws s3/sink.rs`. Look at how other sinks implement templated keys, such as the `console` sink. The fluentd documentation linked provides a reference for the expected format. Note that the implementation should support templating from event fields and include timestamp formatting. Consider how this affects partitioning and ensure backward compatibility.