[Spark 4.2 UT] Umbrella: re-enable disabled Spark 4.2 unit tests (GLUTEN-12569)

Open
#13,181 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
scala

Research direction

Start with sub-issues #13174 through #13180, then inspect the referenced files under gluten-ut/spark42, including VeloxTestSettings.scala and VeloxSQLQueryTestSettings.scala. Use the specific sub-issue to identify a focused group of disabled tests; done means the relevant disables are removed and the gluten-ut/spark42 CI is green.

Written by the indexing model from the issue text.

Description

Umbrella tracking issue for re-enabling the Spark 4.2 unit tests that are currently disabled to keep the gluten-ut/spark42 CI green.

Added in PR apache/gluten#13163 (GLUTEN-12569). 150 UTs are disabled across 7 root-cause themes. Each theme has its own sub-issue with the exact disabled tests, the disable mechanism, and the files updated. As the underlying Gluten/Velox support lands, remove the corresponding .exclude(...) / allowlist / @Ignore / ignore(...) entries and close the sub-issue.

Sub-issues (13)
  • #13174 — Spark 4.2 Storage-Partitioned Join (SPJ) support in Gluten (29 tests)
  • #13175 — Spark 4.2 DSv2 row-level writes (DELETE/MERGE/UPDATE/INSERT) on Gluten path (39 tests)
  • #13176 — Floating-point collection semantics: -0.0/NaN normalization & RESPECT NULLS (12 tests)
  • #13177 — Golden-file drift on Spark 4.2 (TPC-DS plan-stability + SQLQueryTest .sql) (46 tests)
  • #13178 — Velox expression & operator gaps (math/TIME/date/broadcast/hash-agg) (6 tests)
  • #13179 — Spark 4.2 framework/API changes breaking Gluten tests (non-velox) (7 tests)
  • #13180 — File-source & Hive scan (Parquet/ORC ignoreMissingFiles/NullType, bucketed-scan AE, ADD JAR) (11 tests)
  • #13433 — Honor the per-read ignoreMissingFiles option in the native scan (4 tests)
  • #13434 — Parquet NullType (UNKNOWN logical type) interoperability (1 test)
  • #13436 — Spark date_trunc overflow below the int64-microsecond range (1 test)
  • #13438 — collect_list/array_agg RESPECT NULLS drops nulls (1 test)
  • #13439 — NormalizeNaNAndZero is a no-op on Velox: array set ops keep -0.0 (7 tests)
  • #13440 — collect_set does not normalize -0.0/NaN (2 tests)
Summary
# Theme Items Sub-issue
1 Spark 4.2 Storage-Partitioned Join (SPJ) support in Gluten 29 #13174
2 Spark 4.2 DSv2 row-level writes (DELETE/MERGE/UPDATE/INSERT) on Gluten path 39 #13175
3 Floating-point collection semantics: -0.0/NaN normalization & RESPECT NULLS 12 #13176
4 Golden-file drift on Spark 4.2 (TPC-DS plan-stability + SQLQueryTest .sql) 46 #13177
5 Velox expression & operator gaps (math/TIME/date/broadcast/hash-agg) 6 #13178
6 Spark 4.2 framework/API changes breaking Gluten tests (non-velox) 7 #13179
7 File-source & Hive scan (Parquet/ORC ignoreMissingFiles/NullType, bucketed-scan AE, ADD JAR) 11 #13180
Total 150
How these were disabled
  • .exclude(...) in gluten-ut/spark42/.../velox/VeloxTestSettings.scala (most suites, honoring BackendTestSettings.shouldRun)
  • allowlist removal in gluten-ut/spark42/.../velox/VeloxSQLQueryTestSettings.scala (GlutenSQLQueryTestSuite .sql golden tests — that suite ignores the exclude registry)
  • class-level @org.scalatest.Ignore for whole-suite aborts (GlutenBloomFilterFallbackSuite, MiscOperatorSuite, GlutenWholeStageCodegenSuite)
  • test(...)->ignore(...) for one standalone test (GlutenAutoAdjustStageResourceProfileSuite)

All disables are CI-confirmed (local-only false-positives were removed via a validation run with excludes reverted).

Related: PR apache/gluten#13163, GLUTEN-12569.

Dominant language
Scala
Stars
1.6k
Forks
679
Avg merge
4d 16h
Merged PRs (30d)
84

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/gluten

All issues in apache/gluten

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.