[Spark 4.2 UT] Umbrella: re-enable disabled Spark 4.2 unit tests (GLUTEN-12569)

Aperta
#13,181 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
30/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
scala

Direzione di ricerca

Start with sub-issues #13174 through #13180, then inspect the referenced files under gluten-ut/spark42, including VeloxTestSettings.scala and VeloxSQLQueryTestSettings.scala. Use the specific sub-issue to identify a focused group of disabled tests; done means the relevant disables are removed and the gluten-ut/spark42 CI is green.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Umbrella tracking issue for re-enabling the Spark 4.2 unit tests that are currently disabled to keep the gluten-ut/spark42 CI green.

Added in PR apache/gluten#13163 (GLUTEN-12569). 150 UTs are disabled across 7 root-cause themes. Each theme has its own sub-issue with the exact disabled tests, the disable mechanism, and the files updated. As the underlying Gluten/Velox support lands, remove the corresponding .exclude(...) / allowlist / @Ignore / ignore(...) entries and close the sub-issue.

Sub-issues (13)
  • #13174 — Spark 4.2 Storage-Partitioned Join (SPJ) support in Gluten (29 tests)
  • #13175 — Spark 4.2 DSv2 row-level writes (DELETE/MERGE/UPDATE/INSERT) on Gluten path (39 tests)
  • #13176 — Floating-point collection semantics: -0.0/NaN normalization & RESPECT NULLS (12 tests)
  • #13177 — Golden-file drift on Spark 4.2 (TPC-DS plan-stability + SQLQueryTest .sql) (46 tests)
  • #13178 — Velox expression & operator gaps (math/TIME/date/broadcast/hash-agg) (6 tests)
  • #13179 — Spark 4.2 framework/API changes breaking Gluten tests (non-velox) (7 tests)
  • #13180 — File-source & Hive scan (Parquet/ORC ignoreMissingFiles/NullType, bucketed-scan AE, ADD JAR) (11 tests)
  • #13433 — Honor the per-read ignoreMissingFiles option in the native scan (4 tests)
  • #13434 — Parquet NullType (UNKNOWN logical type) interoperability (1 test)
  • #13436 — Spark date_trunc overflow below the int64-microsecond range (1 test)
  • #13438 — collect_list/array_agg RESPECT NULLS drops nulls (1 test)
  • #13439 — NormalizeNaNAndZero is a no-op on Velox: array set ops keep -0.0 (7 tests)
  • #13440 — collect_set does not normalize -0.0/NaN (2 tests)
Summary
# Theme Items Sub-issue
1 Spark 4.2 Storage-Partitioned Join (SPJ) support in Gluten 29 #13174
2 Spark 4.2 DSv2 row-level writes (DELETE/MERGE/UPDATE/INSERT) on Gluten path 39 #13175
3 Floating-point collection semantics: -0.0/NaN normalization & RESPECT NULLS 12 #13176
4 Golden-file drift on Spark 4.2 (TPC-DS plan-stability + SQLQueryTest .sql) 46 #13177
5 Velox expression & operator gaps (math/TIME/date/broadcast/hash-agg) 6 #13178
6 Spark 4.2 framework/API changes breaking Gluten tests (non-velox) 7 #13179
7 File-source & Hive scan (Parquet/ORC ignoreMissingFiles/NullType, bucketed-scan AE, ADD JAR) 11 #13180
Total 150
How these were disabled
  • .exclude(...) in gluten-ut/spark42/.../velox/VeloxTestSettings.scala (most suites, honoring BackendTestSettings.shouldRun)
  • allowlist removal in gluten-ut/spark42/.../velox/VeloxSQLQueryTestSettings.scala (GlutenSQLQueryTestSuite .sql golden tests — that suite ignores the exclude registry)
  • class-level @org.scalatest.Ignore for whole-suite aborts (GlutenBloomFilterFallbackSuite, MiscOperatorSuite, GlutenWholeStageCodegenSuite)
  • test(...)->ignore(...) for one standalone test (GlutenAutoAdjustStageResourceProfileSuite)

All disables are CI-confirmed (local-only false-positives were removed via a validation run with excludes reverted).

Related: PR apache/gluten#13163, GLUTEN-12569.

Lingua principale
Scala
Stelle
1.6k
Fork
685
Merge medio
4g 16h
PR unite (30g)
84

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/gluten

Tutte le issue di apache/gluten

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.