Backport ANSI skipped-operand fix for arithmetic and concat to branch-1.2
维护者通常 2 天内回复
还没有人认领这个 Issue。
评估
调研方向
Read docs/source/contributor-guide/backporting.md and compare the branch-1.2 implementation using NullShortCircuit with the fix proposed in PR 7456, which depends on LazyLogical from PR 7326. Reproduce the ANSI-mode query in the issue and investigate whether wrapping operands gives Spark-compatible results and what performance tradeoff it introduces. Done means proposing and implementing a viable backport approach with correctness and performance addressed.
由索引模型根据 Issue 内容生成。
描述
Describe the bug
branch-1.2 has the ANSI bug from https://github.com/apache/datafusion-comet/issues/7246 for arithmetic and concat: native Comet evaluates an operand on rows where Spark skips it, so a query Spark answers fails with CAST_INVALID_INPUT. For example, with spark.sql.ansi.enabled=true:
CREATE TABLE t(n int, txt string) USING parquet;
INSERT INTO t VALUES (NULL, 'bad'), (3, '6');
SELECT n + CAST(txt AS INT) FROM t; -- Spark: NULL, 9. Comet: CAST_INVALID_INPUT
The fix on main is https://github.com/apache/datafusion-comet/pull/7456 (not merged yet). It cannot be cherry-picked as is: it builds on https://github.com/apache/datafusion-comet/pull/7326 (LazyLogical, speculative evaluation), which branch-1.2 does not have. branch-1.2 does have NullShortCircuit (https://github.com/apache/datafusion-comet/pull/6716), so a backport could wrap the operands without speculation.
This issue tracks whether, and how, to backport the fix to branch-1.2, as docs/source/contributor-guide/backporting.md asks for a correctness fix.
Steps to reproduce
Run the query above on branch-1.2 with ANSI mode on.
Expected behavior
Comet returns the same result as Spark: NULL where n is NULL, without evaluating CAST(txt AS INT) there.
Additional context
- Without speculation, the wrapped form is up to about 2x slower than eager evaluation for cheap operands such as
a * (b - c)on batches with NULLs (benchmark in https://github.com/apache/datafusion-comet/pull/7456). Maintainers may prefer to backport with speculation included, or to accept the slowdown for correctness. - Once https://github.com/apache/datafusion-comet/pull/7456 merges, the backport would be decided per
backporting.md.
Willingness to contribute
I can contribute a fix for this bug independently
- 主要语言
- Scala
- 星标
- 1.3k
- 派生
- 414
- 平均合并
- 3 天 5 小时
- 30 天内合并 PR
- 210
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/datafusion-comet 的其他 Issue
-
requires-triage
难度 2/5 1-3 小时 新手友好度 72/100
apache/datafusion-comet#7583 ·
维护者通常 2 天内回复
-
bug priority:medium
难度 2/5 1-3 小时 新手友好度 73/100
apache/datafusion-comet#7559 · 1 条评论 ·
维护者通常 2 天内回复
-
area:writer bug priority:low
难度 2/5 1-3 小时 新手友好度 78/100
apache/datafusion-comet#7558 ·
维护者通常 2 天内回复
-
enhancement
难度 2/5 1-3 小时 新手友好度 78/100
apache/datafusion-comet#7557 ·
维护者通常 2 天内回复
-
`rand` and `randn` with an integer seed fall back to Spark without a fallback reason可能已有人在做 @galaxytemple 于 3 天前认领。 未关闭requires-triage
难度 2/5 1-3 小时 新手友好度 78/100
apache/datafusion-comet#7479 · 1 条评论 · 已指派 1 人 ·
维护者通常 2 天内回复
查看 apache/datafusion-comet 的全部 Issue
相似的 Issue
-
A-api C-bug S-ready-for-human
难度 2/5 1-3 小时 新手友好度 70/100
ergoplatform/ergo#2932 ·
维护者通常 3 天内回复
-
难度 2/5 1-3 小时 新手友好度 68/100
lichess-org/lila#22454 ·
维护者通常 1 天内回复
-
module: core type: refactor
难度 2/5 1-3 小时 新手友好度 78/100
bigbluebutton/bigbluebutton#26460 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 76/100
chipsalliance/rocket-chip#3845 ·
-
难度 2/5 1-3 小时 新手友好度 76/100