vectordotdev/vector

Allow specifying the object key in the `aws_s3` sink

Open

#6,929 opened on Mar 30, 2021

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Rust (21,837 stars) (2,126 forks)batch import
good first issuesink: aws_s3type: enhancement

Description

Requested by user migrating from fluentd.

Currently, we expose some configuration knobs for the object prefix + key for the aws_s3 sink:

  filename_append_uuid = true # optional, default
  filename_extension = "log" # optional, default
  filename_time_format = "%s" # optional, default
  key_prefix = "date=%F/" # optional, default

However there is no way to directly set the key. Ideally this could be templated from the event (like key = "{{ host }}"). This would affect internal partitioning of events across S3 requests. It is likely we'll still want to support formatting the timestamp as part of that, as well.

Ref: fluentd's s3 output: https://docs.fluentd.org/output/s3#s-3-_object_key_format

Contributor guide

Tech stack
rustaws
Domain
backendclouddata
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Rust programmingAWS S3 basicsVector sink configuration
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50
Research direction
The issue requests adding a configurable key template for AWS S3 sink. Start by examining the existing sink configuration in `src/sinks/aws s3/config.rs` and the key formatting logic in `src/sinks/aws s3/sink.rs`. Look at how other sinks implement templated keys, such as the `console` sink. The fluentd documentation linked provides a reference for the expected format. Note that the implementation should support templating from event fields and include timestamp formatting. Consider how this affects partitioning and ensure backward compatibility.