仓库指标
- 星标
- (64,042 个星标)
- PR 合并指标
- (平均合并 11天 5小时) (30 天内合并 118 个 PR)
描述
Thanos as of the latest version supports vertical query sharding (https://github.com/thanos-io/thanos/pull/5342, https://github.com/thanos-io/thanos/pull/5561). With Sidecar it is done at the Sidecar level i.e. Sidecar retrieves all of the metrics matching the selectors from Prometheus via the remote read interface and then filters out unneeded metrics according to the given shard info. In essence, this means that the amount of data retrieved from Prometheus increases by N where N is the number of shards. This is not ideal on bigger queries because there are a lot of allocations that are immediately freed.
We actually have had an incident where OOM killer killed Prometheus on nodes that have Prometheus+Sidecar running. During one query, RAM usage shot up high, and then OOM killer killed Prometheus as it used the most RAM on that machine.
I propose moving the sharding functionality into the Prometheus remote read interface. I'm not sure but I think that Thanos Sidecar mostly uses that interface so perhaps there wouldn't be any opposition to this?
It should be pretty straightforward - we want to be able to tell Prometheus to only give us X/N of the matching series where X is the shard number and N is the shard count. What do you think?