Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Feature] Support File Index V2 and streaming index writes

Abierto
  • ¿Algo no está bien?
#445 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

@zjw1111 ya está trabajando en esto.

Desde el 19/10/2026.

  • #446 de @zjw1111 — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
cpp
Área
backend, databases

Línea de trabajo

Start by reading the existing File Index container and write path, then compare the V2 layout with the Java reference PR #10101 and the related work in #173. The issue names no source files or tests, so locate the container, writer, plugin interfaces, and spillable output stream before estimating the changes. Done means V1 compatibility remains, V2 reads and writes Java-compatible bytes, and the listed payload, empty-index, and spill-threshold cases are covered; an open linked PR #446 is already working on it.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement
Search before asking
  • I searched in the issues and found no existing issue specifically tracking File Index V2.
Motivation

The V1 File Index container uses 32-bit payload offsets and lengths. The existing write path also materializes serialized index payloads and the complete container in memory before deciding whether to embed the index or write an external file. These limitations make it difficult to support large index artifacts, especially vector indexes.

Java Paimon introduced a V2 container with 64-bit payload ranges and a streaming write interface. Paimon C++ should support the same format while retaining V1 compatibility.

Solution
  • Add V2 container reading and writing, with int64 payload offsets and lengths and a footer/trailer layout aligned with Java Paimon.
  • Support file-index.format-version values 1 and 2, keeping V1 as the default.
  • Add FileIndexWriter::WriteTo() so plugins can write payloads directly to a caller-owned stream. Keep a default implementation based on SerializedBytes() and implement direct streaming output for Lumina.
  • Widen plugin reader offsets and lengths to int64, with plugin-specific size limits checked inside the affected plugins.
  • Integrate a spillable output stream into the data-file index write path, embedding small containers and spilling larger ones to external files without buffering the complete V2 container.
  • Add focused coverage for V1 compatibility, Java-generated V2 golden bytes, large payload ranges, empty indexes, and spill thresholds.

The V2 container's 64-bit ranges do not imply that every index plugin supports payloads larger than 2 GiB; individual plugin serialization formats may retain their own limits.

Anything else?

Java reference: https://github.com/apache/paimon/pull/10101

Related to File Index writing in #173.

Are you willing to submit a PR?
  • I'm willing to submit a PR!
Lenguaje dominante
C++
Estrellas
68
Forks
34
Merge medio
4 d 37 min
PR fusionados (30 d)
35

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/paimon-cpp

Todos los issues de apache/paimon-cpp

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.