Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[lake][paimon] DV tiering skips COMPACT snapshot from the current commit

Abierto
  • ¿Algo no está bien?
#4,582 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@luoyuxia ya está trabajando en esto.

Desde el 10/10/2026.

  • #4583 de @luoyuxia — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
15/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
java
Área
data-engineering

Línea de trabajo

Start in PaimonLakeCommitter, where the commit callback stores a single snapshot ID in a ThreadLocal, then follow that ID into DvTableReadableSnapshotRetriever, which searches for COMPACT snapshots strictly before it. Reproduce with a DV-enabled Paimon primary-key table that produces an APPEND and COMPACT pair in one commit. Done means the committed COMPACT is registered once with the correct tiered and readable offsets, and retention is handled. PR #4583 is already open against this issue, so coordinate with it first.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Search before asking
  • I searched in the issues and found nothing similar.
Fluss version

main (development).

Please describe the bug 🐞

With a Paimon primary-key table using deletion vectors and table.datalake.auto-compaction=true, a single Fluss tiering committable can produce an APPEND snapshot A followed by a COMPACT snapshot C. Paimon invokes the commit callback for both snapshots. PaimonLakeCommitter stores only one snapshot ID in a ThreadLocal, so C overwrites A and is passed to DvTableReadableSnapshotRetriever.

The retriever expects an APPEND ID and searches for a COMPACT strictly before the supplied ID. It therefore skips the just-committed C. On later commits it finds an older COMPACT, but treats the existence of its tiered Fluss metadata as proof that readable offsets are already registered. The retriever returns null, and Fluss registers the new C as tiered-only with KEEP_ALL_PREVIOUS=-1. Historical APPEND offsets are also fetched by APPEND ID, although the paired APPENDs were never registered in Fluss. Repeated commits can leave thousands of COMPACT-only metadata entries and no advancing readable snapshot.

Reproduction outline:

  1. Create a DV-enabled Paimon primary-key table with auto-compaction enabled and tier data from Fluss.
  2. Produce a commit that generates both APPEND A and COMPACT C.
  3. Observe that the final callback ID is C and Fluss records only C without readable offsets.
  4. Repeat. The readable snapshot does not advance and the lake metadata list grows because retention remains -1.

Expected: A newly committed COMPACT is evaluated as a readable-snapshot candidate, with per-bucket readable offsets no greater than that COMPACT's tiered offsets. Historical offsets needed for L0 buckets remain available, and obsolete metadata is eventually pruned.

Actual: The current COMPACT is skipped; subsequent COMPACTs are registered as tiered-only and previous metadata remains indefinitely.

PR #3519 fixes scanning and offset selection for the selected COMPACT, but does not cover this callback/ID mismatch.

Solution

Use the committed COMPACT as the offset-bearing snapshot for the APPEND/COMPACT pair. Both snapshots of one Fluss committable should carry the same prepared fluss-offsets property; verify this in the target Paimon version. Evaluate the current COMPACT inclusively, read its prepared cumulative tiered-offset file before Fluss registration, and resolve a historical APPEND's offsets through its paired registered COMPACT. Register the current COMPACT once with both tiered and, when known, readable offset paths. Retention must use the registered offset carrier's ID, or KEEP_LATEST when no historical carrier is needed. Preserve the conservative fallback if a required source or offset file is unavailable.

Add regression coverage for a real APPEND/COMPACT pair, mixed L0 buckets, COMPACT-only Fluss history, single registration, and retention.

Are you willing to submit a PR?
  • I'm willing to submit a PR!
Lenguaje dominante
Java
Estrellas
2.2k
Forks
646
Merge medio
2 d 28 min
PR fusionados (30 d)
128

Preparar el entorno

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/fluss

Todos los issues de apache/fluss

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.