vectordotdev/vector

Time formats in sinks should be derived from events

Open

#1743 opened on Feb 7, 2020

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Rust (21,837 stars) (2,126 forks)batch import
domain: templatinggood first issuehave: shouldmeta: feedbacksink: aws_s3type: enhancement

Description

It was unclear for me which timestamp is used for making filename. It is not documented anywhere.

My first expectation was to get there timestamp of last event, or maybe timestamp of first event.

But current timestamp is least useful in my point of view — there is a file metadata with creation time. And it doesn't name files as expected when historical data is being processed.

Seems this idea is not new for you https://github.com/timberio/vector/blob/6f290d3e55d78438100b1bd31747c6c4b1630184/src/sinks/aws_s3.rs#L318 but I couldn't find a ticket to track that.

Contributor guide

Tech stack
rust
Domain
backend
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Rust programmingUnderstanding of Vector sinks
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
The issue suggests modifying the AWS S3 sink to derive the timestamp used in filename from the event's timestamp rather than the file creation time. The relevant code is in src/sinks/aws s3.rs line 318, which contains a comment hinting at this future work. To implement, one must understand how events carry timestamps in Vector's data model and how the sink currently obtains the timestamp. Research should focus on exploring the event timestamp availability and modifying the sink to use it, while ensuring backward compatibility.