Native TopK evaluates sort keys for a single-row input, raising false ANSI errors

未关闭
  • 哪里不对？
#7,527 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 2 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
43/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
scala, spark
领域
backend, performance

调研方向

Start by reading the TopK implementation and its tests, then compare its handling of a single-row batch with Spark's TakeOrderedAndProjectExec behavior described in the issue. Reproduce the SQL example with ANSI mode enabled and adaptive execution disabled. Done means the query returns the row in Comet as it does in Spark, without raising ARITHMETIC_OVERFLOW.

由索引模型根据 Issue 内容生成。

描述

bug priority:medium
Describe the bug

With spark.sql.ansi.enabled=true, a TopK (ORDER BY ... LIMIT n) whose whole input is a single row fails in Comet with ARITHMETIC_OVERFLOW when its first sort key overflows on that row, while Spark returns the row.

Spark's TakeOrderedAndProjectExec goes through takeOrdered, which uses Guava's Ordering.leastOf (TopKSelector) per partition and a BoundedPriorityQueue to merge. Both store the first element without a comparison, and sorting a single element compares nothing, so Spark's lazily generated ordering never evaluates any sort key when the input has one row. Comet's native TopK evaluates every sort key over the whole batch.

With two or more rows Spark compares and evaluates the first key for every row, so both engines fail. This is pre-existing and independent of https://github.com/apache/datafusion-comet/pull/7457 , which falls back when a sort key after the first can fail; it was raised in review there.

Steps to reproduce
SET spark.sql.ansi.enabled=true;
SET spark.sql.adaptive.enabled=false;
CREATE TABLE one_row(a int, b int) USING parquet;
INSERT INTO one_row VALUES (1, 2147483647);

SELECT a, b FROM one_row ORDER BY b + 1 LIMIT 10;

Spark returns 1, 2147483647. Comet fails with [ARITHMETIC_OVERFLOW] integer overflow (checked on Spark 4.1 with a CometSqlFileTestSuite fixture, current main plus #7457).

Expected behavior

Same result as Spark. Since it only affects a single-row input, it may be enough to document it rather than fall back for every TopK with a fallible first key.

Additional context

Suggested labels: priority:low, correctness.

主要语言
Scala
星标
1.3k
派生
414
平均合并
3 天 5 小时
30 天内合并 PR
210

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/datafusion-comet 的其他 Issue

查看 apache/datafusion-comet 的全部 Issue

相似的 Issue

更多 Scala Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。