`MoveTowardsLeafNodes` placement implies speculative evaluation, but the docs don't say so
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 75/100
- Issue type
- Documentation
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- backend-api-design, documentation
Research direction
Start with the docs for ExpressionPlacement::MoveTowardsLeafNodes and ScalarUDFImpl::placement, which the issue links. Read how leaf expression extraction and pushdown can evaluate expressions on rows excluded by CASE branches or filters, then document the safety requirement on both APIs. Done when both docs state that expressions with this placement must not fail or have side effects on any input row.
Written by the indexing model from the issue text.
Description
Describe the bug
ExpressionPlacement::MoveTowardsLeafNodes is documented as a performance hint: the expression is cheap and can be pushed down towards the leaves of the plan. When the optimizer acts on that hint, it also runs the expression on rows that the original plan never evaluated it on:
ExtractLeafExpressionsextractsMoveTowardsLeafNodessub-expressions from anywhere inside an expression tree, including the branches ofCASE. In the extraction projection, they are evaluated for every input row, not just the rows the branch selects.PushDownLeafProjectionsthen pushes the extraction projections belowFilternodes, so the expressions also run on rows the filter removes.
So an expression with this placement must be safe to evaluate on rows that a filter or a conditional would exclude: it must not fail on them or have side effects. Neither ExpressionPlacement::MoveTowardsLeafNodes nor ScalarUDFImpl::placement states this. The built-in functions with this placement (get_field, input_file_name, file_row_index) meet the requirement as far as I can tell. A user-defined function that returns MoveTowardsLeafNodes because it is cheap, but fails on some inputs, can make a query fail even though SQL CASE and WHERE semantics say that input is never evaluated.
To Reproduce
The plan below shows how get_field is moved with the default configuration (datafusion.optimizer.enable_leaf_expression_pushdown = true):
CREATE TABLE t AS SELECT * FROM (VALUES
(1, named_struct('a', 1, 'b', 2)),
(-1, named_struct('a', 3, 'b', 4))
) AS v(x, s);
EXPLAIN SELECT CASE WHEN x > 0 THEN s['a'] WHEN x < -100 THEN 0 END FROM t WHERE x <> 5;
logical_plan
01)Projection: CASE WHEN t.x > Int64(0) THEN __datafusion_extracted_1 WHEN t.x < Int64(-100) THEN Int64(0) END AS ...
02)--Filter: t.x != Int64(5)
03)----Projection: get_field(t.s, Utf8("a")) AS __datafusion_extracted_1, t.x
04)------TableScan: t projection=[x, s]
get_field(t.s, 'a') came from the first THEN branch, and it is now evaluated below the filter for every row of t.
Replace s['a'] with a UDF that returns MoveTowardsLeafNodes and fails on some values, for example a checked conversion that fails when its argument is 0. The query then fails on rows where x <= 0 or x = 5, even though the result would never include the UDF's value for those rows.
Expected behavior
Either of these would resolve it:
- Document the requirement (preferred). State on
ExpressionPlacement::MoveTowardsLeafNodesandScalarUDFImpl::placementthat an expression with this placement may be evaluated on rows that enclosing filters or conditional expressions would exclude, so it must not fail or have side effects on any input row. This matches what the optimizer already relies on. - Stop speculative evaluation: don't extract leaf expressions from conditional sub-expressions (
CASEbranches, the right side of short-circuitingAND/OR), and don't push extraction projections below filters when the extracted expression could fail. That would give up much of the benefit of the leaf pushdown optimization, which is mainly about evaluatingget_fieldclose to the scan.
Additional context
The question came up while reviewing #26589, which precomputes MoveTowardsLeafNodes expressions over the whole batch before evaluating CASE branches (https://github.com/apache/datafusion/pull/26589#discussion_r4581490634). That PR relies on the same assumption at the physical level, so documenting the requirement would cover both.
- Dominant language
- Rust
- Stars
- 9.6k
- Forks
- 2.6k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 352
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/datafusion
-
`verify-release-candidate.sh` version check doesn't catch `-dev` versions and is skipped on macOSPossibly taken @vinycalves claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/datafusion#27563 · 1 comment ·
Maintainers usually reply within 1 day
-
`lpad` and `rpad` with a literal length above `i32::MAX` error on a NULL string or fillPossibly taken @TechWizard9999 claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
apache/datafusion#27482 · 1 comment · 1 assignee ·
Maintainers usually reply within 1 day
-
Window constructors panic when the expression list is emptyPossibly taken @abp002 claimed this 4 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/datafusion#27315 · 3 comments · 1 assignee ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/datafusion#27094 · 1 comment ·
Maintainers usually reply within 1 day
-
Substrait consumer doesn't recognise `shift_left` / `shift_right` from functions_arithmetic.yamlPossibly taken @jonasHanhan claimed this 15 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/datafusion#27059 ·
Maintainers usually reply within 1 day
All issues in apache/datafusion
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
tinystream-dev/tinystream#132 ·
-
contract good first issue testing
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Emmyt24/nova-launch#2436 ·
Maintainers usually reply within 1 day