Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[lake][paimon] DV tiering skips COMPACT snapshot from the current commit

已关闭
  • 哪里不对？
#4,582 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

已经有一个关联 PR 被合并了。

  • #4583 来自 @luoyuxia —— 已合并

评估

难度
4/5
预计耗时
3-5 天
新手友好度
15/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
java
领域
data-engineering

调研方向

Start in PaimonLakeCommitter, where the commit callback stores a single snapshot ID in a ThreadLocal, then follow that ID into DvTableReadableSnapshotRetriever, which searches for COMPACT snapshots strictly before it. Reproduce with a DV-enabled Paimon primary-key table that produces an APPEND and COMPACT pair in one commit. Done means the committed COMPACT is registered once with the correct tiered and readable offsets, and retention is handled. PR #4583 is already open against this issue, so coordinate with it first.

由索引模型根据 Issue 内容生成。

描述

Search before asking
  • I searched in the issues and found nothing similar.
Fluss version

main (development).

Please describe the bug 🐞

With a Paimon primary-key table using deletion vectors and table.datalake.auto-compaction=true, a single Fluss tiering committable can produce an APPEND snapshot A followed by a COMPACT snapshot C. Paimon invokes the commit callback for both snapshots. PaimonLakeCommitter stores only one snapshot ID in a ThreadLocal, so C overwrites A and is passed to DvTableReadableSnapshotRetriever.

The retriever expects an APPEND ID and searches for a COMPACT strictly before the supplied ID. It therefore skips the just-committed C. On later commits it finds an older COMPACT, but treats the existence of its tiered Fluss metadata as proof that readable offsets are already registered. The retriever returns null, and Fluss registers the new C as tiered-only with KEEP_ALL_PREVIOUS=-1. Historical APPEND offsets are also fetched by APPEND ID, although the paired APPENDs were never registered in Fluss. Repeated commits can leave thousands of COMPACT-only metadata entries and no advancing readable snapshot.

Reproduction outline:

  1. Create a DV-enabled Paimon primary-key table with auto-compaction enabled and tier data from Fluss.
  2. Produce a commit that generates both APPEND A and COMPACT C.
  3. Observe that the final callback ID is C and Fluss records only C without readable offsets.
  4. Repeat. The readable snapshot does not advance and the lake metadata list grows because retention remains -1.

Expected: A newly committed COMPACT is evaluated as a readable-snapshot candidate, with per-bucket readable offsets no greater than that COMPACT's tiered offsets. Historical offsets needed for L0 buckets remain available, and obsolete metadata is eventually pruned.

Actual: The current COMPACT is skipped; subsequent COMPACTs are registered as tiered-only and previous metadata remains indefinitely.

PR #3519 fixes scanning and offset selection for the selected COMPACT, but does not cover this callback/ID mismatch.

Solution

Use the committed COMPACT as the offset-bearing snapshot for the APPEND/COMPACT pair. Both snapshots of one Fluss committable should carry the same prepared fluss-offsets property; verify this in the target Paimon version. Evaluate the current COMPACT inclusively, read its prepared cumulative tiered-offset file before Fluss registration, and resolve a historical APPEND's offsets through its paired registered COMPACT. Register the current COMPACT once with both tiered and, when known, readable offset paths. Retention must use the registered offset carrier's ID, or KEEP_LATEST when no historical carrier is needed. Preserve the conservative fallback if a required source or offset file is unavailable.

Add regression coverage for a real APPEND/COMPACT pair, mixed L0 buckets, COMPACT-only Fluss history, single registration, and retention.

Are you willing to submit a PR?
  • I'm willing to submit a PR!
主要语言
Java
星标
2.2k
派生
656
平均合并
1 天 21 小时
30 天内合并 PR
127

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 有 Pull Request 模板
  • 没有贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/fluss 的其他 Issue

查看 apache/fluss 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。