CometShuffleManager treats Spark's own shuffles differently from SortShuffleManager

オープン
  • 何か問題がありますか？
#7,488 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
55/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
scala, spark
領域
backend, performance

調査の方向性

Read spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleManager.scala and trace how getWriter, unregisterShuffle, and the IndexShuffleBlockResolver share shuffle state. Check the Spark exchange copy behavior described in the issue and compare cleanup with SortShuffleManager. Done when all three reported differences are addressed; the issue gives no named tests or reproduction.

索引モデルが issue の本文から書いたものです。

説明

area:shuffle bug priority:medium

Describe the bug

CometShuffleManager handles shuffles that Comet does not replace differently from Spark's SortShuffleManager. These shuffles are RDD shuffles and Spark exchanges Comet leaves in place. They are not new, but they reach more applications once the plugin sets CometShuffleManager by default on Spark 4.0 and later (#7336). Found in review of that PR.

  1. Extra row copy in Spark exchanges. ShuffleExchangeExec.needToCopyObjectsBeforeShuffle returns true for any manager that is not a SortShuffleManager, so Spark's own exchanges copy every row before the shuffle.
  2. sortShuffleManager.getWriter entries are never released. For a Spark shuffle dependency, getWriter delegates to the inner sortShuffleManager, which records each map task in its own taskIdMapsForShuffle. unregisterShuffle clears only CometShuffleManager's map and never calls sortShuffleManager.unregisterShuffle, so those entries stay on the executor heap until the application ends.
  3. Blocks migrated in during decommissioning are never deleted. The IndexShuffleBlockResolver is built with a fresh ConcurrentHashMap instead of taskIdMapsForShuffle. The resolver records blocks migrated in during decommissioning in that map, so unregisterShuffle never deletes them.

All three are in spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleManager.scala.

Steps to reproduce

Read the code paths above. No reproduction has been run for any of them.

Expected behavior

A shuffle that Comet does not replace behaves as it does under SortShuffleManager: no extra row copy, and no state left behind after unregisterShuffle.

主要言語
Scala
スター
1.3k
フォーク
414
平均マージ
3日 5時間
マージ済み PR（30日）
210

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/datafusion-comet のほかの issue

apache/datafusion-comet の issue をすべて見る

似ている issue

Scala の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。