prometheus/prometheus

tsdb: [feature request]: detect and stop compacting blocks that result with index file size >64Gi

Open

#5,868 建立於 2019年3月19日

在 GitHub 查看
 (9 留言) (0 反應) (0 負責人)Go (64,042 star) (10,408 fork)batch import
component/tsdbhelp wantedkind/enhancementpriority/P3

描述

We're actually running into this on our largest cluster and it's brought thanos compaction to a standstill. I'm whittling away at cardinality, sharding our scrape targets where possible and backing down our retention policies. Hopefully all of the above will do the trick in the near-term, but it would be great to not have to worry about this in the future.

https://github.com/prometheus/tsdb/blob/e7436e13f07511ee669876905f5147ae1713f5bd/index/index.go#L233

貢獻者指南

技術棧
go
領域
backenddatabaseperformance
議題類型
feature
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
4
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
1-2 days
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
stale
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
mostly clear
前置要求
Go programmingPrometheus TSDB internalscompaction logicindex file size concepts
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
25
研究方向
Investigate the compaction process in Prometheus TSDB, focusing on how blocks are compacted and how the index file size is determined. The key file is index/index.go (line 233 referenced), which likely contains the max size check. Review the 9 comments on the issue for any proposed approaches or workarounds. The goal is to add a check during compaction to detect when a resulting index file would exceed 64GiB and either skip or split that compaction. No linked PRs exist, so the solution is open for contribution.