[EPIC] Support Spark 4.3

オープン
  • 何か問題がありますか？
#7,519 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
github-actions, scala, spark
領域
backend, build-system, ci-cd, testing

調査の方向性

Start with the staged recipe in docs/source/contributor-guide/adding_a_new_spark_version.md and the draft Stage 1 PR, #7518. The issue lists six stages spanning Maven profiles, shims, CI, tests, documentation, and support promotion; begin by checking which stages remain open and their linked work. Done means completing the required stages, including resolving skipped tests and promoting Spark 4.3 to supported.

索引モデルが issue の本文から書いたものです。

説明

enhancement EPIC

Summary

Spark 4.3.0 is released (tag v4.3.0). This tracks bringing up Comet support for it, following the staged recipe in the contributor guide, "Adding Support for a New Spark Version" (docs/source/contributor-guide/adding_a_new_spark_version.md).

Spark 4.3.0 publishes the same Java 17, Scala 2.13.18, Parquet 1.17.0, slf4j 2.0.17, Hadoop 3.5.0 and Arrow 19.0.0 versions as 4.2.0, so the build profile is a copy of spark-4.2.

Stages

  • Stage 1: Maven profile, shims, isSpark43Plus, compile-only lint-java CI row. Draft PR: #7518
  • Stage 2: enable Comet's own JVM test suites under -Pspark-4.3, skipping failures with assume() and a linked issue each.
  • Stage 3: Spark SQL tests (dev/diffs/4.3.0.diff, a spark_4_3 job in ci.yml, compute-changes.py and check-ci-config.py) and plan stability golden files.
  • Stage 4: add 4.3 to the experimental table and jar list in installation.md.
  • Stage 5: fix the skipped tests, one issue or cluster per PR.
  • Stage 6: promote from experimental to supported.

Known Spark 4.3 changes to plan for

Found while doing Stage 1:

  • spark.shuffle.manager must be a BlockingShuffleManager (new in 4.3), and shuffleBlockResolver moved onto it. Comet's shuffle managers need the shim from the Stage 1 PR; the Celeborn integration needs a Celeborn build that targets 4.3, so it should be checked at Stage 2.
  • SpecializedGetters gained getTimestampNTZNanos and getTimestampLTZNanos, for new nanosecond timestamp types. Comet currently throws for them; decide per type whether to fall back or support them.
  • BasePythonRunner passes an Option[PythonWorkerHandle] instead of a pid.
  • SparkEnv.shuffleManager is deprecated in favour of shuffleManagerFor(dependency); there is also a new pipelined shuffle manager kind, which Comet's shuffle planning should not mistake for a blocking one.
  • SimpleCostEvaluator counts local sorts (spark.sql.adaptive.costEvaluator.countLocalSort.enabled), which can change AQE plan choices over Comet plans.
  • An opt-in Arrow cache serializer: tracked separately in #7517.

Not yet decided

  • Which Delta and Iceberg versions to pair with the 4.3 profile. Delta has no 4.3 release; the profile reuses 4.3.1 for compilation only, as spark-4.2 does. Iceberg tests reuse the 4.0 runtime.
主要言語
Scala
スター
1.3k
フォーク
414
平均マージ
3日 5時間
マージ済み PR（30日）
210

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/datafusion-comet のほかの issue

apache/datafusion-comet の issue をすべて見る

似ている issue

Scala の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。