Aggregate spill: List<Dictionary> state whose values repeat within a group still fails under a memory limit

オープン
  • 何か問題がありますか？
#27,240 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
53/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
rust, sql
領域
data-engineering, databases

調査の方向性

Start with #25853 and the spill and replay handling for List<Dictionary> state, especially how SpilledRowSizes estimates referenced values. Reproduce the failure with the provided datafusion-cli -m 24M query, then trace the aggregate spill path and its tests. Done means the query completes under a limit sufficient for one group without causing excessive spill growth when groups share values.

索引モデルが issue の本文から書いたものです。

説明

bug
Describe the bug

An aggregate whose state holds a large dictionary can still fail with Failed to allocate additional ... after #25853 (which fixes #25851), when the dictionary values repeat within a group but not across groups. #25853 sizes a dictionary by the values its keys reference only when fewer than two keys reference each value on average; otherwise it counts the keys only and leaves the dictionary shared, as on main. A dictionary whose values repeat a few times within a group, but whose groups share no values, takes the shared path, and spill and replay batches still carry the whole dictionary.

To Reproduce

Measured with datafusion-cli (release-nonlto) at the head of #25853 (29c666126):

SET datafusion.execution.target_partitions = 1;
SELECT count(*), sum(n) FROM (
  SELECT (v / 10) % 32 AS k,
         cardinality(array_agg(arrow_cast(concat(CAST(v / 10 AS VARCHAR), repeat('x', 100)), 'Dictionary(Int32, Utf8)'))) AS n
  FROM generate_series(1, 5000000) AS t(v)
  GROUP BY (v / 10) % 32);

datafusion-cli -m 24M fails with Failed to allocate additional 24.7 MB for SingleHashAggregateStream[0] with 0.0 B already allocated for this reservation. It passes at -m 64M.

Changing only how often each value repeats in a group (v / 10 in both places):

repeats per value -m 24M
2 passes
10 fails
50 passes

With 1000-byte values and target_partitions = 2, the final aggregate fails at -m 24M (Failed to allocate additional 16.8 MB for FinalHashAggregateStream[0]). With 1000-byte values and target_partitions = 1 over 2M rows, it fails at both 24M and 64M.

Expected behavior

The query completes under a memory limit large enough for one group, as the other repeat counts do.

Additional context

Compacting the dictionary of each spilled chunk would be cheap here, but whether it pays off depends on whether the groups share their values. If they do, each chunk writes its own copy and the spill file grows, which is why #25853 doesn't compact every dictionary (array_agg over 100 distinct 200-byte values spilled 371 MB, against 14.7 MB without compaction). Telling the two cases apart needs the number of distinct values referenced per row, which SpilledRowSizes doesn't track for List<Dictionary>.

Related: #25851, #25853, #26711.

Willingness to contribute

I would like to contribute a fix for this bug, but request guidance from the DataFusion community

主要言語
Rust
スター
9.6k
フォーク
2.6k
平均マージ
3日 4時間
マージ済み PR（30日）
345

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/datafusion のほかの issue

apache/datafusion の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。