vectordotdev/vector

Allow specifying the object key in the `aws_s3` sink

Open

#6.929 geöffnet am 30. März 2021

Auf GitHub ansehen
 (0 Kommentare) (0 Reaktionen) (0 zugewiesene Personen)Rust (21.837 Stars) (2.126 Forks)batch import
good first issuesink: aws_s3type: enhancement

Beschreibung

Requested by user migrating from fluentd.

Currently, we expose some configuration knobs for the object prefix + key for the aws_s3 sink:

  filename_append_uuid = true # optional, default
  filename_extension = "log" # optional, default
  filename_time_format = "%s" # optional, default
  key_prefix = "date=%F/" # optional, default

However there is no way to directly set the key. Ideally this could be templated from the event (like key = "{{ host }}"). This would affect internal partitioning of events across S3 requests. It is likely we'll still want to support formatting the timestamp as part of that, as well.

Ref: fluentd's s3 output: https://docs.fluentd.org/output/s3#s-3-_object_key_format

Contributor Guide

Tech Stack
rustaws
Domain
backendclouddata
Issue Type
feature
SchwierigkeitGeschätzte Implementierungsschwierigkeit für neue Contributors, von 1 für sehr kleine Änderungen bis 5 für Expertenarbeit.
3
Geschätzte ZeitEin grober Zeitrahmen, um zu recherchieren, zu implementieren, zu testen und einen Pull Request vorzubereiten.
half day
AktivitätsstatusWie verfügbar das Issue gerade wirkt: frisch, aktiv, stale, blockiert oder wartet auf Maintainer-Input.
stale
KlarheitWie klar das Issue die erwartete Änderung, Akzeptanzkriterien und den nächsten Schritt erklärt.
clear
Voraussetzungen
Rust programmingAWS S3 basicsVector sink configuration
EinsteigerfreundlichkeitEin Score von 1 bis 100, der schätzt, wie zugänglich dieses Issue für First-Time Contributors ist.
50
Research-Richtung
The issue requests adding a configurable key template for AWS S3 sink. Start by examining the existing sink configuration in `src/sinks/aws s3/config.rs` and the key formatting logic in `src/sinks/aws s3/sink.rs`. Look at how other sinks implement templated keys, such as the `console` sink. The fluentd documentation linked provides a reference for the expected format. Note that the implementation should support templating from event fields and include timestamp formatting. Consider how this affects partitioning and ensure backward compatibility.