sofastack/sofa-jraft

RFC: multi-generation log cache

Open

#854 opened on Jun 23, 2022

View on GitHub
 (9 comments) (1 reaction) (0 assignees)Java (3,406 stars) (1,125 forks)batch import
enhancementhelp wanted

Description

RFC: multi-generation log cache

As discussion in #840 , jraft keeps a cache for latest log entries, it's unlimited and may consume too much memory under write-heavy workload or catching up replication when node restarts.

I think we can abstracts a cache interface and provide a multi-generation cache to replace current one:

  • Gen-1: totaly in memory with limited capacity.
  • Gen-2: Off-heap cache.When Gen-1 overflows, move the LRU(least recently used) item into off-heap cache such as OHC.

Main goals:

  • Abstracts log cache interface.
  • Provide a multi-generation cache implementation.

Contributor guide

Tech stack
java
Domain
backendperformance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
needs maintainer response
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
JavaRaft consensusCaching concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15
Research direction
Review the existing log cache implementation in the codebase and the discussion in issue #840. Understand the current unlimited cache behavior and design a cache interface with multi generation support. Consider using OHC for off heap cache. Propose a detailed design before implementation.