apache/datafusion-ballista

View on GitHub

Apache DataFusion Ballista Distributed Query Engine

Stars: 2.1k
Forks: 321
Open beginner issues: 1
Indexed issues: 146
Avg merge: 1d 23h
Merged PRs (30d): 67

Dominant language: Rust
License: Apache-2.0
Last GitHub push: Sep 18, 2026
Latest indexed: Sep 14, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: good first issue help wanted

Beginner friendly Everything

Open All

144 open issues indexed

AQE replan can swap a broadcast exchange onto the probe side of a CollectLeft join, pinning the join stage to one task Open

apache/datafusion-ballista#2454 · 1 comment · 1 assignee · Opened Sep 14, 2026
Design note: composing multi-partition tasks with plan-level AQE in DataFusion Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

apache/datafusion-ballista#2445 · Opened Sep 12, 2026
Emit the final (post-AQE) physical plan to the event log Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

apache/datafusion-ballista#2422 · Opened Sep 3, 2026
Cancelling a running job leaks its executor vcores and wedges the scheduler under PushStaged Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 65/100

apache/datafusion-ballista#2418 · 1 reaction · Opened Sep 3, 2026
history server event log should capture scheduler id Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 52/100

apache/datafusion-ballista#2413 · Opened Sep 2, 2026
CI: Python client tests do not run on Rust changes and do not test the working tree Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 56/100

apache/datafusion-ballista#2372 · Opened Aug 25, 2026
Python client silently forces the static planner (pinned to ballista 54.0.0 defaults) Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

apache/datafusion-ballista#2371 · 4 comments · Opened Aug 25, 2026
Ballista 55.0.0 Release Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 55/100

apache/datafusion-ballista#2369 · 1 comment · Opened Aug 25, 2026
arrow-ipc-optimizations in ballista-executor does not forward to ballista-core Open

good first issue

Difficulty 2/5 1-3 hours Newbie friendliness 85/100

apache/datafusion-ballista#2368 · Opened Aug 25, 2026
TPCH benchmark fails with `Failed to open partition file at "/var/folders/..../data.arrow"` Open

bug

apache/datafusion-ballista#2367 · 3 comments · 1 reaction · 1 assignee · Opened Aug 25, 2026
Graceful executor shutdown terminates active Flight streams Open

Difficulty 3/5 1-2 days Newbie friendliness 72/100

apache/datafusion-ballista#2357 · Opened Aug 23, 2026
[EPIC] Add public APIs required for scheduler high availability Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

apache/datafusion-ballista#2349 · 4 comments · Opened Aug 20, 2026
Document HA scheduler Open

documentation enhancement

Difficulty 3/5 1-2 days Newbie friendliness 63/100

apache/datafusion-ballista#2347 · 2 comments · Opened Aug 19, 2026
Q10 SF1000 regression after #2315: SortPreservingMergeExec exhausts fair memory pool under AQE default-on Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

apache/datafusion-ballista#2321 · 9 comments · Opened Aug 16, 2026
Allow a stage to start on partially complete inputs instead of waiting for the whole upstream stage Open

enhancement performance

Difficulty 5/5 Over a week Newbie friendliness 35/100

apache/datafusion-ballista#2320 · Opened Aug 16, 2026
Shuffle affinity: prototype a task distribution policy that places consumer tasks where their input already lives Open

enhancement performance

apache/datafusion-ballista#2319 · 1 comment · 1 reaction · 1 assignee · Opened Aug 16, 2026
Buffer shuffle output in memory and spill to disk only when a configured budget is exceeded Open

enhancement performance

Difficulty 5/5 Over a week Newbie friendliness 35/100

apache/datafusion-ballista#2318 · Opened Aug 16, 2026
Add PartialReduce operator / optimization Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 45/100

apache/datafusion-ballista#2314 · Opened Aug 16, 2026
Create a new roadmap for remainder of 2026 Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

apache/datafusion-ballista#2299 · 18 comments · Opened Aug 14, 2026
[DISCUSSION] Restore Flight SQL support as a pluggable query frontend Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

apache/datafusion-ballista#2298 · 2 comments · 4 reactions · Opened Aug 14, 2026
TPC-H SF1000 Q17 stops completing after #2173; task-launch message overflow is misread as executor loss Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

apache/datafusion-ballista#2290 · Opened Aug 12, 2026
Add paging to GET /api/jobs on the scheduler and history server Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

apache/datafusion-ballista#2270 · 2 comments · Opened Aug 9, 2026
Store job event logs in an object store so history survives the scheduler Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

apache/datafusion-ballista#2267 · 1 comment · Opened Aug 9, 2026
Agree and document cross-version compatibility guarantees for Ballista's three versioned surfaces Open

apache/datafusion-ballista#2261 · 3 comments · 1 assignee · Opened Aug 8, 2026
[DISCUSSION] Add support for Spark Connect protocol Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 30/100

apache/datafusion-ballista#2249 · 6 comments · Opened Aug 7, 2026
AQE: coalesce and range-repartition-routing slots can co-exist on the same ExchangeExec Open

Difficulty 4/5 3-5 days Newbie friendliness 55/100

apache/datafusion-ballista#2220 · 1 comment · Opened Aug 3, 2026
Avoid single-task execution for null-aware anti joins (NOT IN subqueries) Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 30/100

apache/datafusion-ballista#2198 · Opened Jul 28, 2026
Default join planning loses NOT IN null semantics in SortMergeJoinExec Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

apache/datafusion-ballista#2193 · 1 reaction · Opened Jul 27, 2026
Add Iceberg integration Open

enhancement

apache/datafusion-ballista#2174 · 10 comments · 1 assignee · Opened Jul 24, 2026
AQE coalesce rule bails on heterogeneous leaf partition counts (per-M subgrouping TODO / Q22 panic guard) Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

apache/datafusion-ballista#2167 · Opened Jul 24, 2026
AQE coalesce rule bails the entire alignment group when any leaf is a broadcast exchange Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

apache/datafusion-ballista#2166 · Opened Jul 24, 2026
Encoded message too large Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

apache/datafusion-ballista#2165 · 2 comments · Opened Jul 24, 2026
Can Ballista do S3 prefetching like Spark? Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

apache/datafusion-ballista#2162 · 1 comment · Opened Jul 23, 2026
Epic: investigate and upstream Spice AI scheduler/executor reliability fixes Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

apache/datafusion-ballista#2158 · 4 comments · Opened Jul 23, 2026
AQE: a replan that reassigns stage ids orphans the sibling leaf stage, and its task completion errors out Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

apache/datafusion-ballista#2150 · Opened Jul 22, 2026
Revisit the hard-coded default of 16 for datafusion.execution.target_partitions Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

apache/datafusion-ballista#2128 · 1 comment · Opened Jul 21, 2026
Broadcast join promotion drops a sort order required by a parent SortMergeJoinExec, silently producing wrong results Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

apache/datafusion-ballista#2065 · 1 comment · Opened Jul 16, 2026
Adaptive (AQE) execution hangs when a replan cancels an in-flight stage Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

apache/datafusion-ballista#2063 · Opened Jul 16, 2026
Distributed execution returns incorrect results for several TPC-DS queries (static planner) Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

apache/datafusion-ballista#2046 · 1 comment · Opened Jul 16, 2026
Job hangs indefinitely instead of failing when all executors are lost Open

bug

Difficulty 5/5 Over a week Newbie friendliness 38/100

apache/datafusion-ballista#2029 · Opened Jul 13, 2026
Hash-join build side OOMs on TPC-H Q18 at SF1000 and does not shrink with target_partitions Open

Difficulty 4/5 3-5 days Newbie friendliness 46/100

apache/datafusion-ballista#2025 · Opened Jul 13, 2026
AQE regresses join-heavy TPC-H queries at SF1000 instead of promoting broadcasts Open

Difficulty 4/5 3-5 days Newbie friendliness 38/100

apache/datafusion-ballista#2024 · 1 comment · Opened Jul 13, 2026
Propagate input ordering across pass-through (non-repartitioning) shuffles Open

enhancement performance

Difficulty 4/5 3-5 days Newbie friendliness 64/100

apache/datafusion-ballista#2006 · Opened Jul 12, 2026
[DISCUSSION] Should Ballista add support for streaming eager execution? Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

apache/datafusion-ballista#2003 · 2 comments · Opened Jul 12, 2026
AQE: immediate intermediate-shuffle reclaim on job success is a no-op under the adaptive planner Open

Difficulty 3/5 1-2 days Newbie friendliness 72/100

apache/datafusion-ballista#1996 · 1 comment · Opened Jul 11, 2026
AQE: switch from streaming aggregation to hash aggregation based on runtime statistics Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 45/100

apache/datafusion-ballista#1989 · Opened Jul 10, 2026
AQE: switch from hash join to sort-merge join based on runtime statistics Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 42/100

apache/datafusion-ballista#1988 · Opened Jul 10, 2026
AQE: handle executor failure on the adaptive execution path Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 42/100

apache/datafusion-ballista#1986 · Opened Jul 10, 2026
Surface collected column statistics to planners via ShuffleReaderExec Open

apache/datafusion-ballista#1960 · 1 assignee · Opened Jul 6, 2026
Reuse repeated exchanges/subplans in the distributed planner (ReuseExchange/ReuseSubquery analog) Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

apache/datafusion-ballista#1953 · Opened Jul 5, 2026

apache/datafusion-ballista

Get new issues in your inbox