yugabyte/yugabyte-db

[DocDB] Allow Docdb flush/compact rate limiter to specify the granularity of enforcement.

Open

#13,404 opened on Jul 22, 2022

View on GitHub
 (2 comments) (0 reactions) (1 assignee)C (1,003 forks)batch import
area/docdbgood first issuekind/enhancementpriority/medium

Repository metrics

Stars
 (8,229 stars)
PR merge metrics
 (Avg merge 17d 21h) (92 merged PRs in 30d)

Description

Jira Link: DB-3034

Description

Rocksdb Rate limiter supports specifying the granularity at which the rate is enforced. This defaults to 100ms

extern RateLimiter* NewGenericRateLimiter(
    int64_t rate_bytes_per_sec,
    int64_t refill_period_us = 100 * 1000,
    int32_t fairness = 10);

However, Docdb doesn't seem to be specifying the granularity and only uses the 100ms default. This can be an issue if the underlying cloud provider enforces their rate limits at a smaller interval.

std::shared_ptr<rocksdb::RateLimiter> CreateRocksDBRateLimiter() {
  if (PREDICT_TRUE((FLAGS_rocksdb_compact_flush_rate_limit_bytes_per_sec > 0))) {
    return std::shared_ptr<rocksdb::RateLimiter>(
      rocksdb::NewGenericRateLimiter(FLAGS_rocksdb_compact_flush_rate_limit_bytes_per_sec));
  }
  return nullptr;
}

A large chunk of write from a flush/compaction may adversely impact our latencies on WAL writes.

Contributor guide

Research direction
Examine the CreateRocksDBRateLimiter function and its callers. Add a new command line flag for refill period and modify the function to use it. Test with different values to ensure proper rate limiting.
Tech stack
cpp
Domain
backenddatabase
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
C++RocksDBDatabase basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65