opensearch-project/OpenSearch

[Remote Translog] Download Translog files concurrently

Open

#10,826 opened on Oct 22, 2023

View on GitHub
 (7 comments) (0 reactions) (1 assignee)Java (1,505 forks)batch import
Storage:Remoteenhancementgood first issue

Repository metrics

Stars
 (8,123 stars)
PR merge metrics
 (Avg merge 5d 9h) (266 merged PRs in 30d)

Description

In failover/primary relocations, we download translog from remote. We need to make it concurrent (still bounded) on a separate threadpool, may be translog_transfer to make that faster . We can use concurrent segment download as a prior work to do the same.

Code pointer : https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java#L205

Concurrent Segment download : https://github.com/opensearch-project/OpenSearch/pull/10519

Contributor guide

Research direction
Study the concurrent segment download implementation in PR #10519 and apply a similar concurrent download pattern to the RemoteFsTranslog translog transfer, using a bounded threadpool and ensuring thread safety.
Tech stack
java
Domain
backend
Issue type
Performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
JavaGit
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60