[VL] Spark 4.2: preserve per-split keyed partition slots for native SPJ (KeyGroupedPartitioning) scans

Aperta
#13,139 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
30/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
scala

Direzione di ricerca

Start by reading shims/spark42/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExecShim.scala, especially filteredPartitions, alongside Spark 4.2's GroupPartitionsExec behavior. Review the gluten-ut/spark42 UT module and the deferred discussion on #13126. Done means preserving one slot per original split or falling back to vanilla Spark, with tests covering partition counts and results on an offloaded scan, timezone, and worker reuse.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Umbrella: GLUTEN-12569 (Spark 4.2 support)

Background

Spark 4.2 restructured Storage-Partitioned Join (SPJ) partitioning. Verified against the Spark jars:

  • GroupPartitionsExec is new in Spark 4.2.0 (absent in 4.1.1). In 4.2 the scan (BatchScanExec) emits ungrouped partitions — one per split, duplicate keys retained — and grouping is deferred to GroupPartitionsExec. In 4.1 the scan produced the key-grouped layout itself.
Problem

The Spark 4.2 shim (shims/spark42/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExecShim.scala, filteredPartitions) still rebuilds the old 4.1 layout via groupBy(InternalRowComparableWrapper(key)), collapsing duplicate-key splits. For keys [A, A, B] it advertises 3 partitions but builds only 2 native RDD partitions; Spark 4.2's GroupPartitionsExec then derives parent indices assuming the ungrouped count → index mismatch / wrong per-key multiplicity.

Scope / why deferred

Only reachable via scans reporting KeyGroupedPartitioning (DSv2 SupportsReportPartitioning). Verified FileScan (Parquet/ORC/CSV) does not implement it, and Delta does not use DSv2 SPJ. The only native consumer is the Iceberg transformer, which is not enabled on Spark 4.2 (dependency unavailable). It is therefore currently dormant / non-triggerable and cannot be behaviorally tested yet.

Required work (when enabling native keyed DSv2 / Iceberg on Spark 4.2)
  1. Preserve one slot per original sorted split (pad filtered-out splits) and let GroupPartitionsExec perform grouping — target layout:
    original [A1,A2,B1]        -> [[A1],[A2],[B1]]
    filter removes A2         -> [[A1],[],[B1]]
    filter removes A1 and A2  -> [[],[],[B1]]
    
    or explicitly fall native keyed scans back to vanilla Spark until (1) is implemented.
  2. Add SPJ tests over an actually offloaded scan with partition-count and result assertions (including a nondefault timezone and worker reuse) — belongs with the gluten-ut/spark42 UT module.
Reference

Deferred from review discussion on #13126.

Lingua principale
Scala
Stelle
1.6k
Fork
685
Merge medio
4g 16h
PR unite (30g)
84

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/gluten

Tutte le issue di apache/gluten

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.