Support sample value updates, modification, and/or accumulation
#13,582 创建于 2024年2月14日
仓库指标
- 星标
- (64,042 个星标)
- PR 合并指标
- (平均合并 11天 5小时) (30 天内合并 118 个 PR)
描述
Proposal
In complex systems it can be necessary to update or amend metric values after-the-fact. This can happen if a metric is emitted by multiple systems that may not report simultaneously, and due to implementation, the user wants the last one to "win". Or, a user wants an initial estimate value recorded ASAP, and then a slow cron job later updates the sample with a more accurate value. Or, a user may want a metric to be a summation of multiple reports of the same metric. These patterns are similar to Out Of Order, for which support was added recently. As a companion feature to OOO, it would be helpful if Prometheus could support metric updating.
We may also consider support for multiple types of updates: replacement is one option, but other metrics may want to accumulate by sum.
We should also consider how this is exposed in the API. Is it a new exposition format grammar, or would users prefer a new API endpoint similar to delete_series?
Would extra care be needed in the case of histograms?
Sketching out a possible approach via scraping/remote write, updates could be accumulated in the WAL/WBL and resolved at compaction time. Updates would probably not be viewable until compaction happens. Subsequent OOO updates would also not be visible until the OOO samples are compacted.