[CI] Detect regressions in native execution and fallback coverage
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- scala, spark
- Domain
- ci-cd, performance, testing
Research direction
Start with the Delta Spark UT gate description in .github/workflows/util/delta-spark-ut/README.md and review the evidence-recording and missing-evidence enforcement in PR #13499. Map the existing execution metrics and test-fork aggregation before designing changes. Done means CI reports per-fork and operation-level coverage, detects missing or duplicate evidence, and fails the specified negative controls without coupling the gate to correctness known-failure lists.
Written by the indexing model from the issue text.
Description
Description
Problem
Correctness-only unit tests can remain green when Gluten stops offloading work and falls back to Spark. The results are still being checked, but the native implementation may no longer be exercised. Plugin registration, enabled configuration, and even a passing native smoke test are insufficient to detect a drop in native execution across the real test workload.
Add CI reporting and regression detection for native versus non-native execution coverage, starting with Delta Spark UT and designing the evidence so it can also support other Gluten test pipelines.
Concrete example: Delta 4.4 partitioned writes
As investigated in #13502, Delta 4.4's writer defaults to physically writing partition columns into Parquet when delta.writePartitionColumnsToParquet is unset. This does not require the separate materializePartitionColumns protocol feature.
Before that PR, Gluten's gate rejected partitioned writes requiring materialization. Under the default settings, those file writes used Delta's JVM writer even with native writes enabled. Reads and other operators could still execute in Velox. Consequently, passing partitioned-table tests and a GlutenDelta command in the plan did not establish native file-write coverage. Enabling native materialization exposed existing native limitations again.
The current Delta known-failures gate compares test outcomes, not how much work ran natively. Removing now-passing failures after fallback can make the baseline look better while native coverage has decreased.
Requested validation
-
Establish minimum native-execution proof from at least one actual workload test in each Spark test JVM/fork expected to exercise Gluten. A missing proof must fail independently of aggregate ratios or correctness baselines. Explicitly document exemptions for forks containing only Spark-free, vanilla-only, or no-work tests; do not infer an exemption merely because no native execution was observed.
-
Record execution evidence with stable test/suite identities and the backend, Spark/Delta versions, and relevant configuration. Report counts and ratios for native, partially native, and fully fallback executions, with explicit handling of Spark-free/no-work tests and intentional disabling.
-
Break coverage down by operation category, especially native scans versus native file writes, rather than relying on one global percentage. Capture fallback reasons and identify which tests or operations lost native coverage.
-
Use final executed plans/runtime evidence where appropriate, including AQE. File writes need evidence of the actual writer used; neither a
GlutenDeltawrapper nor native upstream input operators prove native file writing. Reuse existing metrics/reporting where possible. -
Compare comparable workloads against a base revision or an explicitly reviewed coverage baseline. Account for added/removed tests, matrix/version differences, retries, and duplicate events. Report the numerator and denominator; do not treat a change in raw counts or one aggregate ratio as sufficient evidence by itself.
-
Fail CI for unexpected loss of native coverage according to documented, reviewed expectations/tolerances. Intentional fallbacks remain valid, but coverage reductions require an explicit explanation. Missing or incomplete instrumentation must not silently look like zero applicable work or successful coverage.
-
Keep this gate independent of correctness known-failure lists and flaky quarantine. Refreshing
known-failures.txtmust not automatically accept an offload regression. Canary queries should not count as evidence that the actual workload retained native coverage.
Acceptance criteria
-
Every non-exempt Spark test JVM/fork expected to exercise Gluten provides native execution evidence from the actual workload. A negative control disabling Gluten in one such fork, or dropping its evidence, fails the gate even when other forks still execute natively.
-
CI publishes native/fallback counts, ratios, operation-level breakdowns, and changed test identities alongside correctness results.
-
A negative control that forces previously native partitioned file writes to fall back fails the coverage gate even when results remain correct and native scans/smoke tests still pass.
-
A broad or complete loss of native execution cannot produce a green coverage result.
-
Intentional disabling, unsupported operations, and no-work tests have explicit treatment without requiring every test query to offload.
-
Missing/truncated evidence and duplicate events are detected or handled without inflating coverage; aggregation verifies required shard evidence is present.
-
Baseline comparison and approved coverage changes are documented and covered by positive and negative tests.
Related work and scope
This issue is the single follow-up for native-execution evidence and fallback-regression detection, superseding #13496 and #13499. The earlier PR remains available as an implementation reference; reuse its evidence recording and missing-evidence enforcement where useful rather than maintain two independent validation pipelines.
A real test executing natively proves that Gluten is functioning for that observed execution. One native test somewhere in the overall run does not establish activation in every fork or Spark session. Require native workload evidence per non-exempt Spark test JVM/fork expected to exercise Gluten, and document the coverage boundary; this does not claim that every auxiliary session is correctly configured or that every query must offload. This consolidation does not require porting all of the earlier fixture-level checks. Legitimate fallback and explicitly justified vanilla-only/no-work cases remain supported.
Drafted with assistance from GitHub Copilot (GPT-6 Astra).
Gluten version
main branch
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 679
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 84
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/gluten
-
[CORE] Selecting a cost model the backend has no coster for fails with ClassNotFoundExceptionPossibly taken @LuciferYang claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
bug unit testing velox backend
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/gluten#13463 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
[VL] madvise(WILLNEED) call fails in MmapFileStream because of wrong calculation of fetching lengthMay be free again A pull request for this issue was closed without being merged. Openbug triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 3 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
rchillyard/TableParser#206 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 2 days
-
? - Needs Triage bug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
NVIDIA/cudf-spark#17063 ·
Maintainers usually reply within 2 days
-
bug 🐞 User issue 🚨
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
valkey-io/valkey-glide#7813 · 2 comments ·
Maintainers usually reply within 1 day