[Spark 4.2 UT] Umbrella: re-enable disabled Spark 4.2 unit tests (GLUTEN-12569)

Abierto
#13,181 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
30/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
scala

Línea de trabajo

Start with sub-issues #13174 through #13180, then inspect the referenced files under gluten-ut/spark42, including VeloxTestSettings.scala and VeloxSQLQueryTestSettings.scala. Use the specific sub-issue to identify a focused group of disabled tests; done means the relevant disables are removed and the gluten-ut/spark42 CI is green.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Umbrella tracking issue for re-enabling the Spark 4.2 unit tests that are currently disabled to keep the gluten-ut/spark42 CI green.

Added in PR apache/gluten#13163 (GLUTEN-12569). 150 UTs are disabled across 7 root-cause themes. Each theme has its own sub-issue with the exact disabled tests, the disable mechanism, and the files updated. As the underlying Gluten/Velox support lands, remove the corresponding .exclude(...) / allowlist / @Ignore / ignore(...) entries and close the sub-issue.

Sub-issues (13)
  • #13174 — Spark 4.2 Storage-Partitioned Join (SPJ) support in Gluten (29 tests)
  • #13175 — Spark 4.2 DSv2 row-level writes (DELETE/MERGE/UPDATE/INSERT) on Gluten path (39 tests)
  • #13176 — Floating-point collection semantics: -0.0/NaN normalization & RESPECT NULLS (12 tests)
  • #13177 — Golden-file drift on Spark 4.2 (TPC-DS plan-stability + SQLQueryTest .sql) (46 tests)
  • #13178 — Velox expression & operator gaps (math/TIME/date/broadcast/hash-agg) (6 tests)
  • #13179 — Spark 4.2 framework/API changes breaking Gluten tests (non-velox) (7 tests)
  • #13180 — File-source & Hive scan (Parquet/ORC ignoreMissingFiles/NullType, bucketed-scan AE, ADD JAR) (11 tests)
  • #13433 — Honor the per-read ignoreMissingFiles option in the native scan (4 tests)
  • #13434 — Parquet NullType (UNKNOWN logical type) interoperability (1 test)
  • #13436 — Spark date_trunc overflow below the int64-microsecond range (1 test)
  • #13438 — collect_list/array_agg RESPECT NULLS drops nulls (1 test)
  • #13439 — NormalizeNaNAndZero is a no-op on Velox: array set ops keep -0.0 (7 tests)
  • #13440 — collect_set does not normalize -0.0/NaN (2 tests)
Summary
# Theme Items Sub-issue
1 Spark 4.2 Storage-Partitioned Join (SPJ) support in Gluten 29 #13174
2 Spark 4.2 DSv2 row-level writes (DELETE/MERGE/UPDATE/INSERT) on Gluten path 39 #13175
3 Floating-point collection semantics: -0.0/NaN normalization & RESPECT NULLS 12 #13176
4 Golden-file drift on Spark 4.2 (TPC-DS plan-stability + SQLQueryTest .sql) 46 #13177
5 Velox expression & operator gaps (math/TIME/date/broadcast/hash-agg) 6 #13178
6 Spark 4.2 framework/API changes breaking Gluten tests (non-velox) 7 #13179
7 File-source & Hive scan (Parquet/ORC ignoreMissingFiles/NullType, bucketed-scan AE, ADD JAR) 11 #13180
Total 150
How these were disabled
  • .exclude(...) in gluten-ut/spark42/.../velox/VeloxTestSettings.scala (most suites, honoring BackendTestSettings.shouldRun)
  • allowlist removal in gluten-ut/spark42/.../velox/VeloxSQLQueryTestSettings.scala (GlutenSQLQueryTestSuite .sql golden tests — that suite ignores the exclude registry)
  • class-level @org.scalatest.Ignore for whole-suite aborts (GlutenBloomFilterFallbackSuite, MiscOperatorSuite, GlutenWholeStageCodegenSuite)
  • test(...)->ignore(...) for one standalone test (GlutenAutoAdjustStageResourceProfileSuite)

All disables are CI-confirmed (local-only false-positives were removed via a validation run with excludes reverted).

Related: PR apache/gluten#13163, GLUTEN-12569.

Lenguaje dominante
Scala
Estrellas
1.6k
Forks
685
Merge medio
4 d 16 h
PR fusionados (30 d)
84

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/gluten

Todos los issues de apache/gluten

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.