Bug triage results: 2026-08-31

Bắt đầu với docs/source/contributor-guide/bug_triage.md, sau đó kiểm tra theo mẫu các label và lý do được liệt kê trong issue này, đối chiếu với các issue bị ảnh hưởng được liên kết. Hoàn tất khi đã xác nhận các quyết định triage, trực tiếp sửa các label trên những issue bị ảnh hưởng khi cần và đóng #5601.

Triage pass over the open requires-triage queue, per the project Bug Triage Guide.

• Date: 2026-08-31
• Total issues processed: 77 (73 triaged, 4 skipped, 0 failed)
• Type counts: 24 bugs, 49 enhancements
• Priority counts applied: priority:critical 5, priority:high 2, priority:medium 13, priority:low 4
• Guide: docs/source/contributor-guide/bug_triage.md

Labels have already been applied. A reviewer should spot-check the calls below and close this issue when satisfied; corrections should be made directly on the affected issue.

Note: where an issue already carried a priority label applied by its author, that label was left in place. Per the guide, this process adds a priority label to bugs only.

Bugs

priority:critical

• Checked TIMESTAMP_MILLIS overflow for nested fields and nested-predicate scans is blocked on DataFusion nested-field pruning (#5553)
  • Area labels: area:scan
  • Rationale: nested-field and nested-predicate scans keep overflow-to-NULL where Spark's Math.multiplyExact throws, which is a silent wrong result under decision-tree step 1, even though the fix is blocked on upstream nested-field pruning.
• Match Spark ObjectHashAggregate decimal AVG buffer semantics (#5509)
  • Area labels: area:aggregation
  • Rationale: grouped high-precision decimal AVG returns a different value from Spark with no error when Spark uses ObjectHashAggregateExec, a silent wrong result.
• Match Spark ordering and rank semantics for floating values nested in arrays and structs (#5507)
  • Area labels: area:expressions
  • Rationale: nested -0.0/NaN comparison keys give different ORDER BY and RANK() output from Spark with no error, a silent wrong result.
• Support Spark-compatible Unicode case-insensitive Parquet field matching (#5495)
  • Area labels: area:scan
  • Rationale: filed as a feature request, but the content is a correctness divergence — the native adapter's ASCII-only eq_ignore_ascii_case can miss a present physical column and return SQL NULL instead of the stored value under spark.sql.caseSensitive=false.
• Date-to-timestamp casts can overflow or panic for wide dates (#5456)
  • Area labels: area:expressions
  • Rationale: unchecked multiplication wraps to a bogus timestamp in release builds where Spark throws, which is a silent wrong result; the second reproducer additionally panics.

priority:high

• Native Azure store lets ambient AZURE_* environment variables override or corrupt explicit Hadoop auth config (#5542)
  • Area labels: area:scan
  • Rationale: ambient env credentials silently win over explicitly configured Hadoop auth, so Comet and Spark can resolve different identities for the same table; major functional breakage with a security dimension.
• Iceberg native scan claims schemes it cannot execute; three scheme lists disagree (#5541)
  • Area labels: area:scan
  • Rationale: a gcs/abfs/abfss/wasb/wasbs Iceberg table passes validation, is claimed, and then every task dies with CometNativeException, while stock Spark reads it fine — major functional breakage on supported storage.

priority:medium

• Fix remaining reported issues for arrays of map (#5544)
  • Area labels: area:expressions
  • Rationale: post-merge review of #5452 (merged) finds newly admitted complex literals and a deepNullable cast that changes Slice's nested map type while Slice still declares the original element type; a functional bug with a fallback workaround.
• Iceberg native scan fails queries on tables whose column names are case-distinct to Java but not to Rust (#5540)
  • Area labels: area:scan
  • Rationale: Rust's newer Unicode data folds columns Java keeps distinct, so the query dies with _LEGACY_ERROR_TEMP_2093 — visible breakage with a fallback workaround.
• Native Celeborn shuffle: validate_remote_schema rejects dictionary shapes the shuffle writer can emit (#5536)
  • Area labels: area:shuffle
  • Rationale: local and remote readers disagree about valid frames, so a frame that reads locally fails on Celeborn; a broken feature, and the path is not yet enabled end to end.
• NativeUtil.getNextBatch leaks Arrow structs when importVector fails (#5534)
  • Area labels: area:ffi, area:shuffle
  • Rationale: the third exit from getNextBatch is unguarded, leaking C data and wrapper buffers on every native operator path; resource exhaustion rather than wrong results or an immediate crash.
• unbase64 can fail on rows skipped by LIMIT and semi/anti joins (#5532)
  • Area labels: area:expressions
  • Rationale: batch-at-a-time evaluation turns a query Spark completes into a decode failure; a visible functional bug with an expression-level opt-out.
• Native Celeborn shuffle: the installed Celeborn bootstrap hook can break client creation for the whole executor (#5529)
  • Area labels: area:shuffle
  • Rationale: a Comet-specific bootstrap failure is fatal to all Celeborn client creation on the executor; scoped medium because the native Celeborn path is not enabled end to end yet (see escalations).
• Native Celeborn shuffle: reflectively replacing Celeborn's final fields can release push admission while payloads are in flight (#5528)
  • Area labels: area:shuffle
  • Rationale: reflective assignment to four private final Celeborn fields can release admission early; a broken feature on a path that is not yet enabled.
• Native Celeborn shuffle: default maxFrameBytes of 64 MiB is unreachable, and a large row fails the whole job (#5527)
  • Area labels: area:shuffle
  • Rationale: the two shipped defaults cannot both hold, and the consequence for a large row is a failed job rather than a slow one; configurable, so a workaround exists.
• Comet native broadcast fails under spark.kryo.registrationRequired=true (#5510)
  • Area labels: none
  • Rationale: Array[ChunkedByteBuffer] is unregistered so broadcast throws outright; broken feature, worked around by not setting that Kryo option.
• Avoid object-store cache and registry collisions across backends and configurations (#5502)
  • Area labels: area:scan
  • Rationale: the cache key omits the backend, so a second URL can be served the first URL's store and the DataFusion registry can replace a mapping; a functional bug reachable only with mixed fs.comet.libhdfs.schemes routing.
• Large-offset Arrow vectors from PyArrow UDFs cannot be serialized for broadcast or collect (#5488)
  • Area labels: area:ffi
  • Rationale: Utils.getFieldVector throws Unsupported Arrow Vector for serialize for representations Comet deliberately produces elsewhere; visible failure on a narrow path.
• AQE + DPP + spark.comet.exec.transitionRevert.enabled fails with "SubqueryAdaptiveBroadcastExec does not support the execute() code path" (#5486)
  • Area labels: none
  • Rationale: reversion leaves the plan in a state where PlanAdaptiveDynamicPruningFilters no longer matches, failing the query; gated on a non-default config, so a workaround exists.
• CometExecRule overwrites direct AQE LogicalQueryStage links during replanning (#5482)
  • Area labels: none
  • Rationale: unconditional restoration of originalPlan.logicalLink breaks the correspondence between the current physical root and the active logical stage; a planner defect with no reported wrong-result or crash path.

priority:low

• Surface Parquet TIMESTAMP_MILLIS overflow as a Spark-faithful exception instead of a raw Arrow error (#5517)
  • Area labels: area:scan
  • Rationale: the value is correctly rejected, only the exception type and message diverge from Spark's ArithmeticException("long overflow"); error-fidelity, no wrong results.
• Cancel background batch producers before collecting final plan metrics (#5504)
  • Area labels: area:ffi
  • Rationale: a drop-time metric guard can update counters after the final snapshot, so the reported metrics are incomplete; observability only.
• Explain ObjectHashAggregate fallback when Comet shuffle is disabled (#5500)
  • Area labels: area:aggregation
  • Rationale: the shuffle guard bypasses withFallbackReason, so the strict check can report an unannotated decline; a diagnostics gap, not an execution defect.
• Track provenance of fallback reasons copied through shared expression nodes (#5499)
  • Area labels: area:expressions
  • Rationale: a stale FALLBACK_REASONS tag on a shared node can satisfy reportUnexplainedFallback and hide an unexplained fallback; diagnostics only.

Enhancements

• Derive a native UDF's return type from the library instead of requiring the caller to declare it (#5597)
  • Area labels: area:udf (pre-existing)
  • Rationale: an API ergonomics improvement to the native UDF registration surface; nothing is broken today.
• Align Comet's native UDF surface with SPARK-55278's language-agnostic UDF protocol (#5596)
  • Area labels: area:udf (pre-existing)
  • Rationale: forward-looking alignment with an accepted SPIP targeting Spark 4.3/4.4.
• Rename the .claude directory to a vendor-neutral .ai (#5592)
  • Area labels: none
  • Rationale: repository layout change matching apache/datafusion and apache/datafusion-python; no functional impact.
• next_day and levenshtein fall back to Spark on collated strings (#5591)
  • Area labels: area:expressions
  • Rationale: part of #5572; the current fallback is correct, the request is to route it through the codegen dispatcher instead.
• map_sort falls back to Spark for non-scalar map key types (#5590)
  • Area labels: area:expressions
  • Rationale: part of #5572; widening dispatcher coverage, not a defect.
• map_from_arrays falls back to Spark under mapKeyDedupPolicy=LAST_WIN, unlike map_from_entries (#5589)
  • Area labels: area:expressions
  • Rationale: part of #5572; requests a CodegenDispatchFallback mixin for parity with a sibling serde.
• timestamp_seconds falls back to Spark for decimal, byte and short input (#5588)
  • Area labels: area:expressions
  • Rationale: part of #5572; new input-type coverage.
• abs on interval types falls back to Spark (#5587)
  • Area labels: area:expressions
  • Rationale: part of #5572; new input-type coverage, already flagged good first issue.
• named_struct with duplicate field names falls back to Spark (#5586)
  • Area labels: area:expressions
  • Rationale: part of #5572; documented fallback, request is to dispatch instead.
• translate falls back to Spark by default instead of using the codegen dispatcher like the other string functions (#5585)
  • Area labels: area:expressions
  • Rationale: part of #5572; the Incompatible marking is correct, the ask is dispatcher coverage.
• length / bit_length / octet_length fall back to Spark on binary input (#5584)
  • Area labels: area:expressions
  • Rationale: part of #5572; new input-type coverage.
• arrays_zip falls back to Spark for map element types (#5583)
  • Area labels: area:expressions
  • Rationale: part of #5572; new element-type coverage.
• Array functions fall back to Spark for binary and struct element types (ArraysBase type gate) (#5582)
  • Area labels: area:expressions
  • Rationale: part of #5572; widening a type gate across seven serdes.
• Hash functions fall back to Spark for decimal precision > 18, and sha2 for a non-literal numBits (#5581)
  • Area labels: area:expressions
  • Rationale: part of #5572; a performance/coverage gap, the fallback itself is correct.
• Map lookups with float, collated or complex keys fall back to Spark (map_col[key], element_at) (#5580)
  • Area labels: area:expressions
  • Rationale: part of #5572; the gate exists for real semantic reasons, the ask is to dispatch rather than fall back.
• lpad / rpad with a non-literal pad argument falls back to Spark (#5579)
  • Area labels: area:expressions
  • Rationale: part of #5572; coverage for an ordinary query shape.
• to_csv never runs inside Comet by default, unlike to_json / from_csv / schema_of_csv (#5578)
  • Area labels: area:expressions
  • Rationale: part of #5572; coverage plus a documentation update.
• unix_timestamp on string input falls back to Spark, while to_unix_timestamp already uses the codegen dispatcher (#5577)
  • Area labels: area:expressions
  • Rationale: part of #5572; parity with an equivalent serde.
• round on float/double falls back to Spark, while bround already uses the codegen dispatcher (#5576)
  • Area labels: area:expressions
  • Rationale: part of #5572; parity with an equivalent serde.
• Route unrecognized StaticInvoke and Invoke through the codegen dispatcher instead of falling back (#5575)
  • Area labels: area:expressions
  • Rationale: part of #5572; turns a hard fallback default into in-pipeline execution.
• Codegen dispatcher is unreachable from convert, so serdes that decline there never get a dispatch attempt (#5574)
  • Area labels: area:expressions
  • Rationale: part of #5572; a structural gap in dispatcher reach rather than incorrect output.
• Codegen dispatcher: guard the closure-serialize step so a non-serializable tree falls back cleanly instead of throwing at plan time (#5573)
  • Area labels: area:expressions
  • Rationale: part of #5572; hardening an unexercised failure mode with no reported occurrence.
• [EPIC] Codegen-dispatch coverage audit: expressions that fall back to Spark where the JVM dispatcher would work (#5572)
  • Area labels: area:expressions
  • Rationale: umbrella for the coverage work above; a performance and coverage effort.
• Narrow the invalid-UTF-8 Comet opt-out in the sketch and hll SQL test files (#5571)
  • Area labels: area:scan
  • Rationale: the opt-out is correct but coarser than the problem; test-coverage improvement.
• SQLQueryTestSuite.ignoreList entries skip the Spark baseline, not just Comet (#5570)
  • Area labels: area:ci
  • Rationale: an intentional exclusion that is coarser than intended; narrowing it restores coverage rather than fixing a failure.
• Remove stale dev/diffs test exclusions whose tracking issues are closed (#5569)
  • Area labels: area:ci
  • Rationale: cleanup of exclusions whose cited issues are fixed; restores coverage.
• Native shuffle rejects nested types as hash partitioning keys although the native hasher supports them (#5566)
  • Area labels: area:shuffle
  • Rationale: the current gate is conservative but safe; the ask is to widen it now that nested hashing exists.
• Support useLargeVarTypes in accelerated mapInArrow/mapInPandas (#5555)
  • Area labels: none
  • Rationale: new configuration coverage for the accelerated Python UDF path.
• [FEATURE] Enable direct top-level Variant projection in ordinary native Parquet scans (#5551)
  • Area labels: area:scan
  • Rationale: new functionality, one of five atomic PRs under #5546.
• [FEATURE] Match Spark physical Parquet semantics for projected Variant columns (#5550)
  • Area labels: area:scan
  • Rationale: new functionality under #5546; the Variant path it constrains is not yet shipped.
• [FEATURE] Normalize marked Variant arrays at the native Parquet boundary (#5549)
  • Area labels: area:scan
  • Rationale: new functionality under #5546.
• [FEATURE] Carry Spark VariantType identity through Comet schema serialization (#5548)
  • Area labels: area:scan, area:ffi
  • Rationale: new protobuf and Arrow Field plumbing under #5546.
• [FEATURE] Export complete Arrow Fields through the native FFI boundary (#5547)
  • Area labels: area:ffi
  • Rationale: new FFI capability under #5546; today's exporter is limited, not broken.
• [FEATURE] Native top-level Variant projection from ordinary Parquet (#5546)
  • Area labels: area:scan
  • Rationale: parent tracking issue for re-landing #5407 as five reviewable PRs.
• Native Celeborn shuffle: the remote read path costs up to 4.5x local decode and is unbenchmarked (#5535)
  • Area labels: area:shuffle
  • Rationale: validation is the right call for untrusted bytes; the ask is measurement and optimization.
• Native Celeborn shuffle: add a Celeborn test dependency so the reflection against its internals is verified (#5530)
  • Area labels: area:shuffle
  • Rationale: test-infrastructure improvement.
• Support NullType output types in codegen dispatch (#5525)
  • Area labels: area:expressions
  • Rationale: widening the dispatch type gate for common untyped literals; the current rejection is safe.
• Support Spark's pushed one-field VariantStruct (whole-value pushVariantIntoScan rewrite) in native Parquet scans (#5519)
  • Area labels: area:scan
  • Rationale: new scan capability; the current behavior is an explicit Spark fallback.
• Expose native Parquet scan I/O and read-amplification metrics (#5508)
  • Area labels: area:scan
  • Rationale: new observability counters.
• Narrow strict floating-point admission for corrected scalar sort keys (#5506)
  • Area labels: area:expressions
  • Rationale: admission is conservative but correct after #5469; the ask is to relax it for scalar keys.
• Include days in CalendarInterval hash when Spark 4.3+ is supported (SPARK-58236) (#5498)
  • Area labels: area:expressions
  • Rationale: Comet matches every currently supported Spark version; this is prospective work for a Spark 4.3 profile.
• ci: Prebuild Linux CI images with toolchains and dependencies (#5490)
  • Area labels: area:ci
  • Rationale: build-infrastructure improvement for reliability and setup time.
• Adopt techniques from Spark's ArrowCachedBatchSerializer (SPARK-57268) in Comet's cache format (#5487)
  • Area labels: none
  • Rationale: records design ideas worth adopting from an upstream implementation of the same idea.
• Cached reads that feed Spark operators are slower than Spark's own cache format (#5485)
  • Area labels: none
  • Rationale: a performance gap in a new opt-in feature, not a regression against previously shipped behavior.
• Comet cache decode ignores column projection, making narrow reads of wide cached relations slower than Spark (#5484)
  • Area labels: none
  • Rationale: a design limitation of the new cache serializer from #5051; optimization request.
• Track and remove native Variant compatibility workarounds after upstream fixes (#5477)
  • Area labels: area:scan
  • Rationale: maintenance bookkeeping so intentionally narrow workarounds do not become permanent.
• Remove Variant UTF-16 output rewriting (#5474)
  • Area labels: area:scan
  • Rationale: removal of a correctness-preserving workaround once upstream ordering lands; current behavior is correct.
• Make fair_unified account for spillable consumers (#5465)
  • Area labels: none
  • Rationale: memory-pool accounting improvement matching DataFusion's FairSpillPool; deliberately left out of the focused #5212 fix.
• Track exact SpillWriter page bytes for O(1) memory usage sampling (#5462)
  • Area labels: area:shuffle
  • Rationale: performance follow-up to #5212 finding #12; the correctness fix stays separately scoped.

Escalations to consider

• Native Azure store lets ambient AZURE_* environment variables override or corrupt explicit Hadoop auth config (#5542)
  • The guide lists security vulnerabilities under priority:critical. Ambient environment variables silently overriding an explicitly configured service principal (including the AKS workload-identity webhook case, where the configured client secret is dropped) is an identity-selection issue with a security dimension. Filed priority:high; a reviewer may want to escalate.
• Date-to-timestamp casts can overflow or panic for wide dates (#5456)
  • Matches the guide's escalation trigger in reverse: it both panics (NaiveDate + TimeDelta overflowed) and, in release builds, silently wraps to a wrong timestamp. Filed priority:critical on the wrong-result path.
• Native Celeborn shuffle: the installed Celeborn bootstrap hook can break client creation for the whole executor (#5529)
  • A Comet-specific bootstrap failure is fatal to all Celeborn shuffle on the executor, not just Comet's. Held at priority:medium because the native Celeborn path is not enabled end to end yet (see #5535); escalate to priority:high once it ships enabled.
• Checked TIMESTAMP_MILLIS overflow for nested fields and nested-predicate scans is blocked on DataFusion nested-field pruning (#5553)
  • Filed priority:critical on the silent-wrong-result rule, but the fix is blocked on upstream DataFusion nested-field pruning, so the priority reflects impact rather than actionability.

Skipped — needs more info

• Bug triage results: 2026-08-24 (#5454)
  • A prior triage summary, not a bug or an enhancement, so no type label applies. requires-triage left in place. Note that pass applied no labels (its token lacked label write access), so its 28 issues were re-triaged in this pass. A reviewer should close it.
• Bug triage results: 2026-08-17 (#5385)
  • A prior triage summary awaiting reviewer sign-off; no type label applies and requires-triage was left in place.
• Bug triage results: 2026-08-11 (#5325)
  • A prior triage summary awaiting reviewer sign-off; no type label applies and requires-triage was left in place.
• Bug triage results: 2026-08-03 (#5231)
  • A prior triage summary awaiting reviewer sign-off; no type label applies and requires-triage was left in place.