[EPIC] Support Spark 4.3

Abierto
  • ¿Algo no está bien?
#7,519 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
github-actions, scala, spark
Área
backend, build-system, ci-cd, testing

Línea de trabajo

Start with the staged recipe in docs/source/contributor-guide/adding_a_new_spark_version.md and the draft Stage 1 PR, #7518. The issue lists six stages spanning Maven profiles, shims, CI, tests, documentation, and support promotion; begin by checking which stages remain open and their linked work. Done means completing the required stages, including resolving skipped tests and promoting Spark 4.3 to supported.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement EPIC

Summary

Spark 4.3.0 is released (tag v4.3.0). This tracks bringing up Comet support for it, following the staged recipe in the contributor guide, "Adding Support for a New Spark Version" (docs/source/contributor-guide/adding_a_new_spark_version.md).

Spark 4.3.0 publishes the same Java 17, Scala 2.13.18, Parquet 1.17.0, slf4j 2.0.17, Hadoop 3.5.0 and Arrow 19.0.0 versions as 4.2.0, so the build profile is a copy of spark-4.2.

Stages

  • Stage 1: Maven profile, shims, isSpark43Plus, compile-only lint-java CI row. Draft PR: #7518
  • Stage 2: enable Comet's own JVM test suites under -Pspark-4.3, skipping failures with assume() and a linked issue each.
  • Stage 3: Spark SQL tests (dev/diffs/4.3.0.diff, a spark_4_3 job in ci.yml, compute-changes.py and check-ci-config.py) and plan stability golden files.
  • Stage 4: add 4.3 to the experimental table and jar list in installation.md.
  • Stage 5: fix the skipped tests, one issue or cluster per PR.
  • Stage 6: promote from experimental to supported.

Known Spark 4.3 changes to plan for

Found while doing Stage 1:

  • spark.shuffle.manager must be a BlockingShuffleManager (new in 4.3), and shuffleBlockResolver moved onto it. Comet's shuffle managers need the shim from the Stage 1 PR; the Celeborn integration needs a Celeborn build that targets 4.3, so it should be checked at Stage 2.
  • SpecializedGetters gained getTimestampNTZNanos and getTimestampLTZNanos, for new nanosecond timestamp types. Comet currently throws for them; decide per type whether to fall back or support them.
  • BasePythonRunner passes an Option[PythonWorkerHandle] instead of a pid.
  • SparkEnv.shuffleManager is deprecated in favour of shuffleManagerFor(dependency); there is also a new pipelined shuffle manager kind, which Comet's shuffle planning should not mistake for a blocking one.
  • SimpleCostEvaluator counts local sorts (spark.sql.adaptive.costEvaluator.countLocalSort.enabled), which can change AQE plan choices over Comet plans.
  • An opt-in Arrow cache serializer: tracked separately in #7517.

Not yet decided

  • Which Delta and Iceberg versions to pair with the 4.3 profile. Delta has no 4.3 release; the profile reuses 4.3.1 for compilation only, as spark-4.2 does. Iceberg tests reuse the 4.0 runtime.
Lenguaje dominante
Scala
Estrellas
1.3k
Forks
414
Merge medio
3 d 5 h
PR fusionados (30 d)
210

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-comet

Todos los issues de apache/datafusion-comet

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.