prometheus/prometheus

[Remote Write] WAL watching best-effort start from a checkpoint instead of time.Now

Open

#8,809 opened on May 10, 2021

View on GitHub
 (19 comments) (4 reactions) (0 assignees)Go (64,042 stars) (10,408 forks)batch import
component/remote storagehelp wantednot-as-easy-as-it-looks

Description

Right now remote write starts sending samples with a timestamp after time.Now() when Prometheus started (more or less). On restart or config reload if remote write wasn't able to flush it's pending samples cleanly or was behind for some reason there will be samples missing in the remote storage. If remote write always starts from a checkpoint in some kind of checkpoint file for it's queue name/config hash we can avoid that situation.

see: https://github.com/prometheus/prometheus/pull/7710

Contributor guide

Tech stack
go
Domain
backendobservability
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
GoPrometheus architectureWAL concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
Examine the referenced PR #7710 to understand previous attempts. Investigate the current WAL watching implementation in remote write code (e.g., write handler.go, queue manager.go). Propose a checkpoint file mechanism to persist the last sent sample timestamp.