Native MergeRows can panic or produce wrong masks - arrow 59 `and_not` mishandles sliced arrays

未关闭 适合新手
  • 哪里不对？
#7,596 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 2 天内回复

还没有人认领这个 Issue。

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
76/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
rust, scala, spark
领域
backend, databases

调研方向

从 merge_rows.rs 第 625–626 行附近开始，追踪 MergeRowsStream 如何构建未匹配行的掩码。使用提供的切片数组示例重现 Arrow 59 的行为，然后验证掩码是否与 and(a, not(b)) 一致，以及原生 MergeRows 是否不再 panic；issue 建议将两处 and_not 调用替换作为临时解决方法。

由索引模型根据 Issue 内容生成。

描述

requires-triage
Describe the bug

MergeRowsStream builds the "not matched" masks with arrow::compute::kernels::boolean::and_not (merge_rows.rs:625-626). In arrow 59.x and_not mixes up the offsets of its inputs when one of them is a slice (apache/arrow-rs#10698, fixed in apache/arrow-rs#10699 for 60.0.0, not backported). Comet main resolves arrow to 59.3.2, so the bug is still there.

After the full outer join one marker column can be a slice with a non-zero offset. Then the task either panics with offset + len out of bounds, or gets wrong masks with no error and applies inserts and deletes to the wrong rows. We hit the panic on Iceberg MERGE with spark.comet.exec.mergeRows.enabled=true (Comet 1.2.0, Spark 4.0.2); with the flag off the jobs pass.

Steps to reproduce
// arrow = "=59.3.2"
use arrow::array::{Array, BooleanArray};
use arrow::compute::kernels::boolean::{and, and_not, not};

fn mask(len: usize, f: fn(usize) -> bool) -> BooleanArray {
    BooleanArray::from((0..len).map(f).collect::<Vec<_>>())
}

fn main() {
    let left = mask(3000, |i| i % 3 != 0).slice(0, 1000);
    let right = mask(2000, |i| i % 5 == 0).slice(1000, 1000);
    let expected = and(&left, &not(&right).unwrap()).unwrap();
    let actual = and_not(&left, &right).unwrap();
    let wrong = (0..1000).filter(|&i| actual.value(i) != expected.value(i)).count();
    println!("wrong values: {wrong} of 1000"); // 59.x: 533, 60.1: 0
}
Expected behavior

and_not(a, b) gives the same result as and(a, not(b)), and native MergeRows does not panic.

Additional context

Until Comet moves to arrow 60 (#7249), replacing the two calls with and(a, &not(b)?) fixes it.

Willingness to contribute

I can contribute a fix for this bug independently

主要语言
Scala
星标
1.3k
派生
418
平均合并
2 天 23 小时
30 天内合并 PR
158

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/datafusion-comet 的其他 Issue

查看 apache/datafusion-comet 的全部 Issue

相似的 Issue

更多 Scala Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。