[EPIC] Support Spark 4.3

Aperta
  • Qualcosa non va?
#7,519 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
github-actions, scala, spark
Ambito
backend, build-system, ci-cd, testing

Direzione di ricerca

Start with the staged recipe in docs/source/contributor-guide/adding_a_new_spark_version.md and the draft Stage 1 PR, #7518. The issue lists six stages spanning Maven profiles, shims, CI, tests, documentation, and support promotion; begin by checking which stages remain open and their linked work. Done means completing the required stages, including resolving skipped tests and promoting Spark 4.3 to supported.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement EPIC

Summary

Spark 4.3.0 is released (tag v4.3.0). This tracks bringing up Comet support for it, following the staged recipe in the contributor guide, "Adding Support for a New Spark Version" (docs/source/contributor-guide/adding_a_new_spark_version.md).

Spark 4.3.0 publishes the same Java 17, Scala 2.13.18, Parquet 1.17.0, slf4j 2.0.17, Hadoop 3.5.0 and Arrow 19.0.0 versions as 4.2.0, so the build profile is a copy of spark-4.2.

Stages

  • Stage 1: Maven profile, shims, isSpark43Plus, compile-only lint-java CI row. Draft PR: #7518
  • Stage 2: enable Comet's own JVM test suites under -Pspark-4.3, skipping failures with assume() and a linked issue each.
  • Stage 3: Spark SQL tests (dev/diffs/4.3.0.diff, a spark_4_3 job in ci.yml, compute-changes.py and check-ci-config.py) and plan stability golden files.
  • Stage 4: add 4.3 to the experimental table and jar list in installation.md.
  • Stage 5: fix the skipped tests, one issue or cluster per PR.
  • Stage 6: promote from experimental to supported.

Known Spark 4.3 changes to plan for

Found while doing Stage 1:

  • spark.shuffle.manager must be a BlockingShuffleManager (new in 4.3), and shuffleBlockResolver moved onto it. Comet's shuffle managers need the shim from the Stage 1 PR; the Celeborn integration needs a Celeborn build that targets 4.3, so it should be checked at Stage 2.
  • SpecializedGetters gained getTimestampNTZNanos and getTimestampLTZNanos, for new nanosecond timestamp types. Comet currently throws for them; decide per type whether to fall back or support them.
  • BasePythonRunner passes an Option[PythonWorkerHandle] instead of a pid.
  • SparkEnv.shuffleManager is deprecated in favour of shuffleManagerFor(dependency); there is also a new pipelined shuffle manager kind, which Comet's shuffle planning should not mistake for a blocking one.
  • SimpleCostEvaluator counts local sorts (spark.sql.adaptive.costEvaluator.countLocalSort.enabled), which can change AQE plan choices over Comet plans.
  • An opt-in Arrow cache serializer: tracked separately in #7517.

Not yet decided

  • Which Delta and Iceberg versions to pair with the 4.3 profile. Delta has no 4.3 release; the profile reuses 4.3.1 for compilation only, as spark-4.2 does. Iceberg tests reuse the 4.0 runtime.
Lingua principale
Scala
Stelle
1.3k
Fork
414
Merge medio
3g 5h
PR unite (30g)
210

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-comet

Tutte le issue di apache/datafusion-comet

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.