[Bug] Cluster rewrite closes the shared IOManager, breaking later spills of the same write

Open Beginner friendly
#10,066 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
java, spark
Domain
data-engineering

Research direction

Start at Sorter.close() and the clusterRewrite path, then trace ownership of the shared IOManager and the sorter's buffer. Verify completion by exercising bucketed append-only clustering that spills across multiple groups without FileNotFoundException, and confirm the input RecordReaderIterator is closed.

Written by the indexing model from the issue text.

Description

Search before asking
  • I searched in the issues and found nothing similar.
Paimon version

master

Compute Engine

Flink / Spark (bucketed append-only table clustering)

Minimal reproduce step

On a bucketed append-only table with clustering enabled, trigger a compaction whose write instance clusters more than one (partition, bucket) group, and give the sort buffer enough data to spill (a small write-buffer-size / small page size, or enough rows). The first group's clustering succeeds. The second one throws java.io.FileNotFoundException: .../paimon-io-*/....channel (No such file or directory) as soon as its sort buffer spills.

What doesn't meet your expectations?

Sorter.close() closes the IOManager it was handed. That IOManager belongs to the write and is shared by everything the write does, and its spill directories are created once, when the manager is constructed. Closing it deletes those directories, so after the first clustering round any later spill of the same write instance fails with FileNotFoundException.

Expected: clustering a table whose write spills across more than one group does not fail.

Anything else?

Fix direction: Sorter must not close the caller-owned IOManager. The sorter's own resources are released by buffer.clear(), which deletes its spill channels through the buffer's private SpillChannelManager, so nothing sorter-owned leaks. Separately, the sorter's input RecordReaderIterator was never closed by clusterRewrite; close it in Sorter.close().

Are you willing to submit a PR?
  • I'm willing to submit a PR!
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 14h
Merged PRs (30d)
491

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/paimon

All issues in apache/paimon

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.