prometheus/prometheus

Reduce the impact of remote write resharding

Open

#7,230 opened on May 8, 2020

View on GitHub
 (27 comments) (0 reactions) (0 assignees)Go (10,408 forks)batch import
component/remote storagehelp wantednot-as-easy-as-it-lookspriority/P3

Repository metrics

Stars
 (64,042 stars)
PR merge metrics
 (Avg merge 11d 5h) (118 merged PRs in 30d)

Description

Right now resharding, especially sharding up is very disruptive to throughput. The resharding process drains all queues, which takes a significant amount of time if the remote endpoint is having issues. This will block new samples from being appended while they queues clear, and one slow shard can cause throughput to drop significantly.

Instead of waiting for all shards to flush to remote storage we could send them into the new shards that are being created, being sure to rebalance them into the appropriate shard.

Contributor guide

Research direction
Examine the remote write queue and shard management in the codebase. Understand the current resharding logic that causes throughput drops. Propose an alternative approach to rebalance samples without draining all queues.
Tech stack
go
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Go
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30