influxdata/telegraf

Configuration option to drop oldest or newest data in the buffer when buffer is full

Open

#15,217 opened on Apr 24, 2024

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Go (9,892 stars) (4,161 forks)batch import
feature requesthelp wantedsize/m

Description

Use Case

Similar to https://github.com/influxdata/telegraf/issues/15208

(accommodating temporary high-write bursts to and influxdb instance)

Sometimes the newest data is more important to keep than the oldest data (if your computer failed, would you care more about what happened right before the failure or 5 minutes before the failure?)

Sometimes the oldest data is more important to keep than the newest data (ex: which events happened first in this very large burst of events?)

Expected behavior

I expected to see a configuration option like "buffer_drop_strategy" which would default to "drop_oldest".

Actual behavior

No configuration option to specify which metrics (newest or oldest) are dropped from the buffer.

Additional info

No response

Contributor guide

Tech stack
go
Domain
backenddata
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Basic understanding of Telegraf's output buffer configuration
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65
Research direction
Examine the buffer implementation in Telegraf, likely in the 'internal/buffer' package or similar, to understand how metrics are dropped when the buffer is full (e.g., drop oldest logic). Identify where the configuration is parsed (e.g., output plugin config) and add a new 'buffer drop strategy' option. Implement the logic to drop either oldest or newest based on the option, ensuring it is tested and documented.