yugabyte/yugabyte-db

[DocDB] Enable printing out why each RocksDB flush is being done

Closed

#25,615 opened on Jan 14, 2025

View on GitHub
 (3 comments) (0 reactions) (1 assignee)C (8,229 stars) (1,003 forks)batch import
area/docdbgood first issuekind/enhancementpriority/medium

Description

Jira Link: DB-14868

Description

We have had two incidents now where inability to figure out why RocksDB flushes were occurring was key and extremely hard to figure out.

This task is to build instrumentation to make this easier in the future.

That is, create a runtime-settable gflag that turns on logging messages that for each flush describe why it is being done.

Reasons might include:

  • to stay under the global MemTable limit
  • to stay under the local MemTable limit
  • by request of a flush table RPC call
  • flush requested by DoCleanupIntentFiles

The above is almost certainly not a complete list. Ideally this task instruments all the calls to flush.

Although not ideal, it's probably acceptable for the reason log message to be given earlier than the log message announcing the flush itself as long as they can be correlated.

Ideally backport this to all versions in use.

Issue Type

kind/enhancement

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.

Contributor guide

Tech stack
cppc
Domain
databaseobservability
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
blocked
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
C++ programmingRocksDB internalsYugabyteDB build systemgflags configuration
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
Identify all call sites where RocksDB flush is triggered (e.g., in src/rocksdb/db/db impl.cc). Add a runtime gflag (e.g., 'rocksdb flush reason logging') that enables logging the reason before each flush. The reason can be passed through a new argument to flush functions. Correlate with existing flush log messages. See comments on the issue for possible reasons list.
[DocDB] Enable printing out why each RocksDB flush is being done · yugabyte/yugabyte-db#25615 | Good First Issue