vectordotdev/vector

Time formats in sinks should be derived from events

Open

#1,743 创建于 2020年2月7日

在 GitHub 查看
 (4 评论) (0 反应) (0 负责人)Rust (21,837 star) (2,126 fork)batch import
domain: templatinggood first issuehave: shouldmeta: feedbacksink: aws_s3type: enhancement

描述

It was unclear for me which timestamp is used for making filename. It is not documented anywhere.

My first expectation was to get there timestamp of last event, or maybe timestamp of first event.

But current timestamp is least useful in my point of view — there is a file metadata with creation time. And it doesn't name files as expected when historical data is being processed.

Seems this idea is not new for you https://github.com/timberio/vector/blob/6f290d3e55d78438100b1bd31747c6c4b1630184/src/sinks/aws_s3.rs#L318 but I couldn't find a ticket to track that.

贡献者指南

技术栈
rust
领域
backend
议题类型
feature
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
3
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
1-3 hours
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
stale
清晰度议题是否清楚说明期望改动、验收标准和下一步。
mostly clear
前置要求
Rust programmingUnderstanding of Vector sinks
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
30
研究方向
The issue suggests modifying the AWS S3 sink to derive the timestamp used in filename from the event's timestamp rather than the file creation time. The relevant code is in src/sinks/aws s3.rs line 318, which contains a comment hinting at this future work. To implement, one must understand how events carry timestamps in Vector's data model and how the sink currently obtains the timestamp. Research should focus on exploring the event timestamp availability and modifying the sink to use it, while ensuring backward compatibility.