vectordotdev/vector

Allow specifying the object key in the `aws_s3` sink

Open

#6 929 ouverte le 30 mars 2021

Voir sur GitHub
 (0 commentaires) (0 réactions) (0 assignés)Rust (21 837 stars) (2 126 forks)batch import
good first issuesink: aws_s3type: enhancement

Description

Requested by user migrating from fluentd.

Currently, we expose some configuration knobs for the object prefix + key for the aws_s3 sink:

  filename_append_uuid = true # optional, default
  filename_extension = "log" # optional, default
  filename_time_format = "%s" # optional, default
  key_prefix = "date=%F/" # optional, default

However there is no way to directly set the key. Ideally this could be templated from the event (like key = "{{ host }}"). This would affect internal partitioning of events across S3 requests. It is likely we'll still want to support formatting the timestamp as part of that, as well.

Ref: fluentd's s3 output: https://docs.fluentd.org/output/s3#s-3-_object_key_format

Guide contributeur

Stack technique
rustaws
Domaine
backendclouddata
Type d'issue
feature
DifficultéDifficulté estimée pour un nouveau contributeur, de 1 pour un très petit changement à 5 pour un travail expert.
3
Temps estiméFourchette de temps approximative pour investiguer, implémenter, tester et préparer une pull request.
half day
Statut d'activitéDisponibilité apparente de l'issue : fraîche, active, ancienne, bloquée ou en attente d'un mainteneur.
stale
ClartéClarté avec laquelle l'issue explique le changement attendu, les critères d'acceptation et la prochaine étape.
clear
Prérequis
Rust programmingAWS S3 basicsVector sink configuration
Accessibilité débutantScore de 1 à 100 estimant l'accessibilité de cette issue pour un premier contributeur.
50
Direction de recherche
The issue requests adding a configurable key template for AWS S3 sink. Start by examining the existing sink configuration in `src/sinks/aws s3/config.rs` and the key formatting logic in `src/sinks/aws s3/sink.rs`. Look at how other sinks implement templated keys, such as the `console` sink. The fluentd documentation linked provides a reference for the expected format. Note that the implementation should support templating from event fields and include timestamp formatting. Consider how this affects partitioning and ensure backward compatibility.