[Bug] hll_sketch_agg, hll_union and hll_union_agg write compact sketch bytes where Spark writes updatable ones
维护者通常 2 天内回复
还没有人认领这个 Issue。
评估
调研方向
Read the affected files listed in the issue: native/spark-expr/src/agg_funcs/hll_sketch.rs, hll_sketch_agg.rs, hll_union_agg.rs, native/spark-expr/src/hll_scalar.rs, and the CometHll serdes under spark/src/main/spark-4.x/org/apache/comet/serde/. Start by tracing how these expressions serialize LIST and SET sketches and compare the bytes with Spark's output. Done means the returned bytes use Spark's updatable layout and match Spark for the reproduced query.
由索引模型根据 Issue 内容生成。
描述
Describe the bug
On Spark 4.x, Comet's native hll_sketch_agg, hll_union and hll_union_agg return sketch bytes that differ from Spark's when the sketch is still in LIST or SET mode (a small number of distinct values).
Spark returns HllSketch.toUpdatableByteArray, which is documented as the "updatable binary representation". The datasketches crate (0.6.0) has only a compact serialize(). For LIST and SET it sets COMPACT_FLAG and writes the compact layout (list.rs:131, hash_set.rs:163).
Both layouts decode to the same sketch, so hll_sketch_estimate and hll_union read either one correctly and the estimates match. The bytes themselves still differ, so:
- any query that compares, groups by, hashes, or stores the raw sketch bytes gets a different result under Comet;
- DataSketches-Java
HllSketch.writableWraprejects Comet's output because it only accepts updatable images (HllSketch.java:251-254in 6.2.0).
This is why these three expressions still need spark.comet.expression.<name>.allowIncompatible=true after the audit in https://github.com/apache/datafusion-comet/issues/7396 .
Steps to reproduce
-- Spark 4.1, few distinct values -> LIST mode
SELECT hex(hll_sketch_agg(col)) FROM VALUES (1), (2), (3) AS t(col);
Run it once with Comet disabled and once with spark.comet.expression.HllSketchAgg.allowIncompatible=true. The hex strings differ: Comet sets COMPACT_FLAG and writes only the coupons, while Spark writes the updatable layout. hll_sketch_estimate returns 3 in both cases.
Comparison experiment (DataSketches-Java 6.1.1 and 6.2.0 vs Comet with datasketches 0.6.0, 4030 cases): https://github.com/tlfua/datafusion-comet/actions/runs/42348769378 . Summary: https://github.com/apache/datafusion-comet/issues/7396#issuecomment-6703600973 .
Expected behavior
Comet writes the same updatable layout as Spark, so the bytes match and these expressions can run without the opt-in.
Possible fixes:
- Add an updatable serialization to datasketches-rust (https://github.com/apache/datasketches-rust ) and use it in Comet.
- Re-encode LIST/SET sketches into the updatable layout in Comet before returning them.
Additional context
Surfaced by the HLL audit for https://github.com/apache/datafusion-comet/issues/7396 . Affected Comet files: native/spark-expr/src/agg_funcs/hll_sketch.rs, hll_sketch_agg.rs, hll_union_agg.rs, native/spark-expr/src/hll_scalar.rs and the CometHll* serdes under spark/src/main/spark-4.x/org/apache/comet/serde/.
Separately, the estimate after merging partial sketches can differ by up to 0.81% (hll_sketch_agg with more than one partition) because SET-to-HLL promotion during a union visits coupons in a different order. Spark's own estimate for the same input changes with partitioning by a similar or larger amount.
Willingness to contribute
I would be willing to contribute a fix for this bug with guidance from the DataFusion community
- 主要语言
- Scala
- 星标
- 1.3k
- 派生
- 414
- 平均合并
- 3 天 5 小时
- 30 天内合并 PR
- 210
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/datafusion-comet 的其他 Issue
-
requires-triage
难度 2/5 1-3 小时 新手友好度 72/100
apache/datafusion-comet#7583 ·
维护者通常 2 天内回复
-
bug priority:medium
难度 2/5 1-3 小时 新手友好度 73/100
apache/datafusion-comet#7559 · 1 条评论 ·
维护者通常 2 天内回复
-
area:writer bug priority:low
难度 2/5 1-3 小时 新手友好度 78/100
apache/datafusion-comet#7558 ·
维护者通常 2 天内回复
-
enhancement
难度 2/5 1-3 小时 新手友好度 78/100
apache/datafusion-comet#7557 ·
维护者通常 2 天内回复
-
`rand` and `randn` with an integer seed fall back to Spark without a fallback reason可能已有人在做 @galaxytemple 于 3 天前认领。 未关闭requires-triage
难度 2/5 1-3 小时 新手友好度 78/100
apache/datafusion-comet#7479 · 1 条评论 · 已指派 1 人 ·
维护者通常 2 天内回复
查看 apache/datafusion-comet 的全部 Issue
相似的 Issue
-
A-api C-bug S-ready-for-human
难度 2/5 1-3 小时 新手友好度 70/100
ergoplatform/ergo#2932 ·
维护者通常 3 天内回复
-
难度 2/5 1-3 小时 新手友好度 68/100
lichess-org/lila#22454 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 76/100
chipsalliance/rocket-chip#3845 ·
-
难度 2/5 1-3 小时 新手友好度 76/100
-
难度 1/5 1 小时以内 新手友好度 86/100
维护者通常 13 天内回复