Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Feature] Support writing ARRAY<BLOB> fields

Aperta
  • Qualcosa non va?
#389 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp, java, python
Ambito
data-engineering, databases

Direzione di ricerca

Start with the ARRAY reader work from #387/#388, then trace BlobFormatWriter, schema validation, the data-evolution write path, and append/data-evolution compaction. Use the BLOB file specification as the format reference. Done means validation, null and descriptor handling, writing, compaction, round-trip tests, and Java/Python compatibility tests all pass.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement
Search before asking
  • I searched in the issues and found nothing similar.
Motivation

Java Paimon (apache/paimon#8181) and PyPaimon can write top-level ARRAY<BLOB> columns to dedicated .blob files. #387 (PR #388) adds read-only ARRAY<BLOB> support to Paimon C++, but C++ table creation, writes, and append compaction still reject these fields. As a result, C++ users cannot write ordered collections of multimodal objects, such as the frames or images of one sample, to a Paimon table.

Solution

Add Java-compatible write support for top-level ARRAY<BLOB> fields declared with blob-field:

  • Schema validation: Match Java SchemaValidation and ColumnDirectiveUtils:
    • allow ARRAY<BLOB> only as a top-level field declared with blob-field;
    • reject it for blob-descriptor-field, partition keys, and tables without data evolution;
    • reject BLOB elements nested more deeply, such as ARRAY<ARRAY<BLOB>>.
  • Blob format writer: Extend BlobFormatWriter to encode each ARRAY<BLOB> row as the nested payload defined in the BLOB file spec. The payload contains:
    • array magic 1094861634;
    • version 1;
    • element count;
    • concatenated element data;
    • Delta-Varint element length index;
    • index length.
  • Null handling: Distinguish null arrays (outer length -1), empty arrays (element count 0), null elements (element length -1), and empty elements (element length 0).
  • Element sources: Support both inline raw bytes and BlobDescriptor elements. Descriptor elements are materialized from their referenced storage when written, as Java does.
  • Write path: Route ARRAY<BLOB> fields to dedicated blob files in the data-evolution write path, and remove the write-side rejection for them.
  • Compaction: Support ARRAY<BLOB> in append and data-evolution compaction, including placeholder (-2) handling across field groups.
  • Tests:
    • add unit tests for payload encoding, null/empty cases, and error handling;
    • add round-trip tests with the C++ reader from #387;
    • add Java/Python cross-compatibility tests, with C++ writing and Java/Python reading.
Anything else?
  • Depends on #387 / #388 for the reader side.
  • MAP<K, BLOB> write support and ARRAY<BLOB> in primary-key managed BLOB storage (#204) are out of scope and can be tracked separately.
  • No public API changes are expected. The storage format follows the existing Java spec.
Are you willing to submit a PR?
  • I'm willing to submit a PR!
Lingua principale
C++
Stelle
65
Fork
29
Merge medio
2g 5h
PR unite (30g)
77

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/paimon-cpp

Tutte le issue di apache/paimon-cpp

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.