Release a task's vcores as its partitions finish

Open
#2,905 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Dandandan is already working on this.

Since Nov 20, 2026.

  • #2904 by @Dandandan — open

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
18/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
rust

Research direction

This is a design-scale change across the scheduler and executor, and an open pull request (#2904) already carries work toward it — start by reading that PR to see the intended shape. The issue names the pieces to trace: the vcore bind_one reservation, the shuffle writers that would report finished partitions, pull-mode semaphore permits in poll_work, and push-mode running task status plus the scheduler's refund of a reservation. Read the open questions before starting, since memory-pool sizing and extra-small-task binding are unresolved; "done" is a session setting that gates the behaviour, with both pull and push modes reporting released vcores and the terminal status returning the remainder.

Written by the indexing model from the issue text.

Description

enhancement

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

A task that packs N partitions is charged N vcores (bind_one) and holds all of them until its slowest partition ends. When the partitions of a task are skewed, for example one large file or one hot key next to many small ones, most of those vcores sit idle for most of the task. Meanwhile other stages and other jobs wait for an executor with free vcores, so the cluster looks full while a good part of it is doing nothing.

This got more visible when ballista.scheduler.max_partitions_per_task started defaulting to 0 (unbounded): a single task now commonly holds every free vcore of an executor.

Describe the solution you'd like

Hand a partition's vcore back as soon as that partition finishes, while the rest of the task keeps running:

  • The shuffle writers report each input partition they finish draining.
  • A task needs min(running partitions, vcores) vcores and keeps at least one until it ends.
  • Pull mode: the task releases the finished partitions' semaphore permits, so the next poll_work reports them as free.
  • Push mode: the executor reports the released vcores in a running task status, and the scheduler refunds them from the task's reservation. The terminal status returns whatever is left.
  • A session setting turns it off.

Collapse stages hold a single vcore and pipelined jobs reserve vcores per job, so neither is affected.

Describe alternatives you've considered

  • Lowering ballista.scheduler.max_partitions_per_task. Smaller tasks free their vcores sooner, but every stage then pays for more tasks whether its partitions are skewed or not, and a task of several partitions still waits for its slowest one.
  • Splitting skewed partitions at planning time. This helps where the skew is known up front, but not when it only shows at run time.

Additional context

Open questions:

  • A task's memory pool is sized by the vcores it was charged and is not shrunk when vcores are released, so the task pools on an executor can temporarily add up to more than its memory.
  • Vcores freed mid-stage are bound to that stage's remaining partitions as extra small tasks, which may cost more than it gains for stages without skew.
  • Pipelined jobs could release from their per-job reservation in a similar way, as a follow-up.
Dominant language
Rust
Stars
2.2k
Forks
348
Avg merge
2d 4h
Merged PRs (30d)
166

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/datafusion-ballista

All issues in apache/datafusion-ballista

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.