JuliaParallel/Dagger.jl

Wrap inputs in Chunk only once, or not at all

Open

#200 opened on Jan 28, 2021

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Julia (706 stars) (86 forks)batch import
data movementhelp wantedperformance

Description

In #199 I've fixed a "bug" where non-thunk inputs are serialized during the thunk RPC call, which is silly for large inputs. That fix assumes that inputs are large enough to wrap in a Chunk, which is not always true. It also re-wraps inputs in Chunks multiple times, which can incur extra overhead. We should make a dynamic decision whether to wrap in a Chunk, and if we wrap, wrap just once.

Contributor guide

Tech stack
None
Domain
backendperformance
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Understand Dagger's thunk executionRead issue #199
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
45
Research direction
Review PR #199 to understand the current chunk wrapping logic. Examine the thunk RPC call path to identify where inputs are wrapped. Propose a condition to decide when to wrap and ensure wrapping happens only once.