Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[lake][paimon] DV tiering skips COMPACT snapshot from the current commit

Chiusa
  • Qualcosa non va?
#4,582 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Una pull request collegata è già stata integrata.

  • #4583 di @luoyuxia — integrata

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
15/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
java
Ambito
data-engineering

Direzione di ricerca

Start in PaimonLakeCommitter, where the commit callback stores a single snapshot ID in a ThreadLocal, then follow that ID into DvTableReadableSnapshotRetriever, which searches for COMPACT snapshots strictly before it. Reproduce with a DV-enabled Paimon primary-key table that produces an APPEND and COMPACT pair in one commit. Done means the committed COMPACT is registered once with the correct tiered and readable offsets, and retention is handled. PR #4583 is already open against this issue, so coordinate with it first.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Search before asking
  • I searched in the issues and found nothing similar.
Fluss version

main (development).

Please describe the bug 🐞

With a Paimon primary-key table using deletion vectors and table.datalake.auto-compaction=true, a single Fluss tiering committable can produce an APPEND snapshot A followed by a COMPACT snapshot C. Paimon invokes the commit callback for both snapshots. PaimonLakeCommitter stores only one snapshot ID in a ThreadLocal, so C overwrites A and is passed to DvTableReadableSnapshotRetriever.

The retriever expects an APPEND ID and searches for a COMPACT strictly before the supplied ID. It therefore skips the just-committed C. On later commits it finds an older COMPACT, but treats the existence of its tiered Fluss metadata as proof that readable offsets are already registered. The retriever returns null, and Fluss registers the new C as tiered-only with KEEP_ALL_PREVIOUS=-1. Historical APPEND offsets are also fetched by APPEND ID, although the paired APPENDs were never registered in Fluss. Repeated commits can leave thousands of COMPACT-only metadata entries and no advancing readable snapshot.

Reproduction outline:

  1. Create a DV-enabled Paimon primary-key table with auto-compaction enabled and tier data from Fluss.
  2. Produce a commit that generates both APPEND A and COMPACT C.
  3. Observe that the final callback ID is C and Fluss records only C without readable offsets.
  4. Repeat. The readable snapshot does not advance and the lake metadata list grows because retention remains -1.

Expected: A newly committed COMPACT is evaluated as a readable-snapshot candidate, with per-bucket readable offsets no greater than that COMPACT's tiered offsets. Historical offsets needed for L0 buckets remain available, and obsolete metadata is eventually pruned.

Actual: The current COMPACT is skipped; subsequent COMPACTs are registered as tiered-only and previous metadata remains indefinitely.

PR #3519 fixes scanning and offset selection for the selected COMPACT, but does not cover this callback/ID mismatch.

Solution

Use the committed COMPACT as the offset-bearing snapshot for the APPEND/COMPACT pair. Both snapshots of one Fluss committable should carry the same prepared fluss-offsets property; verify this in the target Paimon version. Evaluate the current COMPACT inclusively, read its prepared cumulative tiered-offset file before Fluss registration, and resolve a historical APPEND's offsets through its paired registered COMPACT. Register the current COMPACT once with both tiered and, when known, readable offset paths. Retention must use the registered offset carrier's ID, or KEEP_LATEST when no historical carrier is needed. Preserve the conservative fallback if a required source or offset file is unavailable.

Add regression coverage for a real APPEND/COMPACT pair, mixed L0 buckets, COMPACT-only Fluss history, single registration, and retention.

Are you willing to submit a PR?
  • I'm willing to submit a PR!
Lingua principale
Java
Stelle
2.2k
Fork
656
Merge medio
2g 21h
PR unite (30g)
97

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/fluss

Tutte le issue di apache/fluss

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.