Release a task's vcores as its partitions finish
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 18/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- rust
- Domain
- distributed-systems, performance
Research direction
This is a design-scale change across the scheduler and executor, and an open pull request (#2904) already carries work toward it — start by reading that PR to see the intended shape. The issue names the pieces to trace: the vcore bind_one reservation, the shuffle writers that would report finished partitions, pull-mode semaphore permits in poll_work, and push-mode running task status plus the scheduler's refund of a reservation. Read the open questions before starting, since memory-pool sizing and extra-small-task binding are unresolved; "done" is a session setting that gates the behaviour, with both pull and push modes reporting released vcores and the terminal status returning the remainder.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
A task that packs N partitions is charged N vcores (bind_one) and holds all of them until its slowest partition ends. When the partitions of a task are skewed, for example one large file or one hot key next to many small ones, most of those vcores sit idle for most of the task. Meanwhile other stages and other jobs wait for an executor with free vcores, so the cluster looks full while a good part of it is doing nothing.
This got more visible when ballista.scheduler.max_partitions_per_task started defaulting to 0 (unbounded): a single task now commonly holds every free vcore of an executor.
Describe the solution you'd like
Hand a partition's vcore back as soon as that partition finishes, while the rest of the task keeps running:
- The shuffle writers report each input partition they finish draining.
- A task needs
min(running partitions, vcores)vcores and keeps at least one until it ends. - Pull mode: the task releases the finished partitions' semaphore permits, so the next
poll_workreports them as free. - Push mode: the executor reports the released vcores in a running task status, and the scheduler refunds them from the task's reservation. The terminal status returns whatever is left.
- A session setting turns it off.
Collapse stages hold a single vcore and pipelined jobs reserve vcores per job, so neither is affected.
Describe alternatives you've considered
- Lowering
ballista.scheduler.max_partitions_per_task. Smaller tasks free their vcores sooner, but every stage then pays for more tasks whether its partitions are skewed or not, and a task of several partitions still waits for its slowest one. - Splitting skewed partitions at planning time. This helps where the skew is known up front, but not when it only shows at run time.
Additional context
Open questions:
- A task's memory pool is sized by the vcores it was charged and is not shrunk when vcores are released, so the task pools on an executor can temporarily add up to more than its memory.
- Vcores freed mid-stage are bound to that stage's remaining partitions as extra small tasks, which may cost more than it gains for stages without skew.
- Pipelined jobs could release from their per-job reservation in a similar way, as a follow-up.
- Dominant language
- Rust
- Stars
- 2.2k
- Forks
- 348
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 166
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/datafusion-ballista
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/datafusion-ballista#2885 · 1 reaction ·
Maintainers usually reply within 1 day
-
Executor's default work dir is deleted at startup and recreated without `TempDir`'s permissionsPossibly taken @aetherxeg-source claimed this 16 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/datafusion-ballista#2767 ·
Maintainers usually reply within 1 day
-
Iceberg: DROP TABLE fails on a registered table its catalog can't loadPossibly taken @NoahKusaba claimed this 15 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 43/100
apache/datafusion-ballista#2882 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
apache/datafusion-ballista#2881 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
apache/datafusion-ballista#2872 · 1 comment ·
Maintainers usually reply within 1 day
All issues in apache/datafusion-ballista
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
antithesishq/bombadil#473 ·
Maintainers usually reply within 2 days
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
libracoder/kiri#669 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 5 days
-
autofixer: follow-up
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
tursodatabase/turso#12217 ·
Maintainers usually reply within 1 day
-
Declare the CMS schema library in the optional recipients extraPossibly taken A pull request linked to this issue is open or already merged. Openarea:python in-progress
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
yhay81/pyropdf#1005 · 2 comments ·
Maintainers usually reply within 1 day