Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

JVM columnar shuffle panics on a struct column with a NullType field

未关闭 适合新手
  • 哪里不对？
#7,062 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
72/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
java, rust
领域
data-engineering

调研方向

从 native/shuffle/src/spark_unsafe/row.rs 开始，查看两个按字段组织的函数 append_struct_fields_field_major 和 append_nested_struct_fields_field_major，它们在 DataType::Null 上会触发 unreachable!。对比同一文件中 append_field 已有的 Null 分支，了解每行的 null 是如何追加的。当 issue 中的三个 SQL 复现返回与 Spark 相同的行，并且有一个回归测试覆盖顶层 struct 和含 null 字段的嵌套 struct 时，即完成。

由索引模型根据 Issue 内容生成。

描述

area:shuffle bug priority:high requires-triage
Describe the bug

A shuffle that runs on Comet's JVM columnar shuffle fails the query when one of its columns is a struct with a NullType field, such as named_struct('x', a, 'n', null). The row-to-Arrow conversion panics:

org.apache.comet.CometNativeException: internal error: entered unreachable code: Unsupported data type of struct field: Null
        at datafusion_comet_shuffle::spark_unsafe::row::append_struct_fields_field_major
        at datafusion_comet_shuffle::spark_unsafe::row::append_columns
        at datafusion_comet_shuffle::spark_unsafe::row::process_sorted_row_partition
        at comet::execution::jni_api::write_sorted_file
        at comet::execution::jni_api::Java_org_apache_comet_Native_writeSortedFileNative

Spark returns the rows. Both field-major struct paths in native/shuffle/src/spark_unsafe/row.rs lack a DataType::Null arm and fall through to unreachable!. append_struct_fields_field_major handles a top-level struct column, and append_nested_struct_fields_field_major handles a struct inside a struct:

https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/native/shuffle/src/spark_unsafe/row.rs#L1042-L1047

https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/native/shuffle/src/spark_unsafe/row.rs#L582-L587

append_field in the same file already has the arm. The JVM columnar shuffle's type check accepts NullType at any depth, so nothing keeps these schemas away from the converter, and Null is the only type that check accepts that the two functions are missing.

https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/native/shuffle/src/spark_unsafe/row.rs#L272-L275

With the default configuration this hits every round-robin repartition of such a column, because native round robin is off by default and those shuffles run on JVM columnar shuffle. It also hits a hash shuffle that runs on JVM columnar shuffle, for example under spark.comet.shuffle.mode=jvm. A hash shuffle over a native child runs on native shuffle and works. Arrays and maps that hold nulls (array(null), map(k, null), array(named_struct('x', a, 'n', null))) also work.

Steps to reproduce
CREATE TABLE t (a INT) USING parquet;
INSERT INTO t SELECT CAST(id AS INT) FROM range(1000);

-- top-level struct column
SELECT /*+ REPARTITION(4) */ a, named_struct('x', a, 'n', null) AS st FROM t;

-- struct inside a struct
SELECT /*+ REPARTITION(4) */ a, named_struct('s', named_struct('x', a, 'n', null)) AS st FROM t;

-- hash partitioning on JVM columnar shuffle
SET spark.comet.shuffle.mode=jvm;
SELECT a, named_struct('x', a, 'n', null) AS st FROM t DISTRIBUTE BY a;

All three fail with the panic above. I reproduced them on c3316fbe39 with the Spark 4.1 profile, and the code involved is unchanged on main at 3a4a159409.

Expected behavior

The queries should return the same rows as Spark. Every value of a NullType field is null, so a DataType::Null arm in both functions that appends one null per row, as append_field does, should be enough. A regression test with a top-level and a nested struct would cover both paths.

Additional context

#6977 proposes merging the two field-major functions, which would make an arm like this one change instead of two.

主要语言
Scala
星标
1.3k
派生
391
平均合并
2 天 10 小时
30 天内合并 PR
421

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/datafusion-comet 的其他 Issue

查看 apache/datafusion-comet 的全部 Issue

相似的 Issue

更多 Scala Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。