apple/foundationdb

DD: Build teams in batches to avoid slow task from DD

Open

#2,102 opened on Sep 12, 2019

View on GitHub
 (3 comments) (0 reactions) (0 assignees)C++ (13,912 stars) (1,266 forks)batch import
data distributiongood first issue

Description

The current DD build all required teams in one big atomic block.

When there are a lot of teams to build, say over 6000 teams, DD will take tens of seconds (say 20 seconds) to build all teams without yield. This leads to slow task which makes DD non-responsive to other requests, such as heart beat. Slow tasks can lead to problems: such as cluster controller can mark a live DD as dead, and data balance can be blocked.

The solution to this issue is to break the non-stopping block of buildTeams() into smaller asynchronous blocks. When a lot of teams are to build, buildTeams() will build some of them (say 100 teams) and yield. Later, buildTeams() build another batch of teams.

This issue is related to Issue #2096.

Contributor guide

Tech stack
cpp
Domain
databaseperformance
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
C++ programmingFoundationDB architectureasynchronous programming concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35
Research direction
Examine the buildTeams() function in the Data Distributor (DD) component. Look at how teams are currently built atomically. Consider splitting the work into batches of 100 teams using asynchronous operations. Check issue #2096 for related context. Ensure that the batching does not introduce data races or deadlocks. Test with large numbers of teams to verify performance improvement.
DD: Build teams in batches to avoid slow task from DD · apple/foundationdb#2102 | Good First Issue