[CI] Detect regressions in native execution and fallback coverage
I maintainer di solito rispondono entro 3 giorni
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 25/100
- Tipo di issue
- Funzionalità
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Stack tecnologico
- scala, spark
- Ambito
- ci-cd, performance, testing
Direzione di ricerca
Start with the Delta Spark UT gate description in .github/workflows/util/delta-spark-ut/README.md and review the evidence-recording and missing-evidence enforcement in PR #13499. Map the existing execution metrics and test-fork aggregation before designing changes. Done means CI reports per-fork and operation-level coverage, detects missing or duplicate evidence, and fails the specified negative controls without coupling the gate to correctness known-failure lists.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Description
Problem
Correctness-only unit tests can remain green when Gluten stops offloading work and falls back to Spark. The results are still being checked, but the native implementation may no longer be exercised. Plugin registration, enabled configuration, and even a passing native smoke test are insufficient to detect a drop in native execution across the real test workload.
Add CI reporting and regression detection for native versus non-native execution coverage, starting with Delta Spark UT and designing the evidence so it can also support other Gluten test pipelines.
Concrete example: Delta 4.4 partitioned writes
As investigated in #13502, Delta 4.4's writer defaults to physically writing partition columns into Parquet when delta.writePartitionColumnsToParquet is unset. This does not require the separate materializePartitionColumns protocol feature.
Before that PR, Gluten's gate rejected partitioned writes requiring materialization. Under the default settings, those file writes used Delta's JVM writer even with native writes enabled. Reads and other operators could still execute in Velox. Consequently, passing partitioned-table tests and a GlutenDelta command in the plan did not establish native file-write coverage. Enabling native materialization exposed existing native limitations again.
The current Delta known-failures gate compares test outcomes, not how much work ran natively. Removing now-passing failures after fallback can make the baseline look better while native coverage has decreased.
Requested validation
-
Establish minimum native-execution proof from at least one actual workload test in each Spark test JVM/fork expected to exercise Gluten. A missing proof must fail independently of aggregate ratios or correctness baselines. Explicitly document exemptions for forks containing only Spark-free, vanilla-only, or no-work tests; do not infer an exemption merely because no native execution was observed.
-
Record execution evidence with stable test/suite identities and the backend, Spark/Delta versions, and relevant configuration. Report counts and ratios for native, partially native, and fully fallback executions, with explicit handling of Spark-free/no-work tests and intentional disabling.
-
Break coverage down by operation category, especially native scans versus native file writes, rather than relying on one global percentage. Capture fallback reasons and identify which tests or operations lost native coverage.
-
Use final executed plans/runtime evidence where appropriate, including AQE. File writes need evidence of the actual writer used; neither a
GlutenDeltawrapper nor native upstream input operators prove native file writing. Reuse existing metrics/reporting where possible. -
Compare comparable workloads against a base revision or an explicitly reviewed coverage baseline. Account for added/removed tests, matrix/version differences, retries, and duplicate events. Report the numerator and denominator; do not treat a change in raw counts or one aggregate ratio as sufficient evidence by itself.
-
Fail CI for unexpected loss of native coverage according to documented, reviewed expectations/tolerances. Intentional fallbacks remain valid, but coverage reductions require an explicit explanation. Missing or incomplete instrumentation must not silently look like zero applicable work or successful coverage.
-
Keep this gate independent of correctness known-failure lists and flaky quarantine. Refreshing
known-failures.txtmust not automatically accept an offload regression. Canary queries should not count as evidence that the actual workload retained native coverage.
Acceptance criteria
-
Every non-exempt Spark test JVM/fork expected to exercise Gluten provides native execution evidence from the actual workload. A negative control disabling Gluten in one such fork, or dropping its evidence, fails the gate even when other forks still execute natively.
-
CI publishes native/fallback counts, ratios, operation-level breakdowns, and changed test identities alongside correctness results.
-
A negative control that forces previously native partitioned file writes to fall back fails the coverage gate even when results remain correct and native scans/smoke tests still pass.
-
A broad or complete loss of native execution cannot produce a green coverage result.
-
Intentional disabling, unsupported operations, and no-work tests have explicit treatment without requiring every test query to offload.
-
Missing/truncated evidence and duplicate events are detected or handled without inflating coverage; aggregation verifies required shard evidence is present.
-
Baseline comparison and approved coverage changes are documented and covered by positive and negative tests.
Related work and scope
This issue is the single follow-up for native-execution evidence and fallback-regression detection, superseding #13496 and #13499. The earlier PR remains available as an implementation reference; reuse its evidence recording and missing-evidence enforcement where useful rather than maintain two independent validation pipelines.
A real test executing natively proves that Gluten is functioning for that observed execution. One native test somewhere in the overall run does not establish activation in every fork or Spark session. Require native workload evidence per non-exempt Spark test JVM/fork expected to exercise Gluten, and document the coverage boundary; this does not claim that every auxiliary session is correctly configured or that every query must offload. This consolidation does not require porting all of the earlier fixture-level checks. Legitimate fallback and explicitly justified vanilla-only/no-work cases remain supported.
Drafted with assistance from GitHub Copilot (GPT-6 Astra).
Gluten version
main branch
- Lingua principale
- Scala
- Stelle
- 1.6k
- Fork
- 685
- Merge medio
- 4g 16h
- PR unite (30g)
- 84
Preparare l'ambiente
Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di apache/gluten
-
[CORE] Selecting a cost model the backend has no coster for fails with ClassNotFoundExceptionForse già presa @LuciferYang l’ha presa 1 giorno fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
I maintainer di solito rispondono entro 3 giorni
-
bug unit testing velox backend
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
apache/gluten#13463 · 1 commento ·
I maintainer di solito rispondono entro 3 giorni
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 82/100
I maintainer di solito rispondono entro 3 giorni
-
[VL] madvise(WILLNEED) call fails in MmapFileStream because of wrong calculation of fetching lengthForse di nuovo libera Una pull request per questa issue è stata chiusa senza essere unita. Apertabug triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
I maintainer di solito rispondono entro 3 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 3 giorni
Tutte le issue di apache/gluten
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
rchillyard/TableParser#206 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 2 giorni
-
? - Needs Triage bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
NVIDIA/cudf-spark#17063 ·
I maintainer di solito rispondono entro 2 giorni
-
bug 🐞 User issue 🚨
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
valkey-io/valkey-glide#7813 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno