Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Feature] Support writing MAP<K, BLOB> fields

Aperta
  • Qualcosa non va?
#415 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@SteNicholas ci sta già lavorando.

Dal 2/10/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

enhancement
Search before asking
  • I searched in the issues and found nothing similar.
Motivation

Java Paimon (apache/paimon#8635, with key types extended in apache/paimon#8963 and apache/paimon#9057) and PyPaimon can write top-level MAP<K, BLOB> columns to dedicated .blob files. #283 (PR #278) adds read-only MAP<K, BLOB> support to Paimon C++, and #389 (PR #392) adds write support for ARRAY<BLOB>, but C++ table creation (TableSchema::Create) and writes (FileStoreWrite::Create) still reject MAP<K, BLOB> fields through BlobUtils::ValidateContainerBlobWriteSchema. As a result, C++ users cannot write keyed collections of multimodal objects, such as the images of one sample keyed by camera name, to a Paimon table.

Solution

Add Java-compatible write support for top-level MAP<K, BLOB> fields declared with blob-field:

  • Schema validation: Keep the rules consistent with Java and with ARRAY<BLOB> (#392):
    • allow MAP<K, BLOB> only as a top-level field declared with blob-field;
    • reject it for blob-descriptor-field, blob-view-field, partition keys, and tables without data evolution;
    • reject unsupported key types and BLOB values nested more deeply, such as MAP<K, ARRAY<BLOB>> or ARRAY<MAP<K, BLOB>>.
  • Blob format writer: Extend BlobFormatWriter to encode each MAP<K, BLOB> row as the nested payload defined in the BLOB file spec and decoded by the C++ reader from #278. The payload contains:
    • map magic 1296188226;
    • version 1;
    • entry count;
    • concatenated key data;
    • concatenated BLOB data;
    • Delta-Varint key length index;
    • Delta-Varint BLOB length index;
    • key index length and BLOB index length.
  • Key encoding: Match the Java key encodings for the key types supported by the C++ reader:
    • TINYINT, SMALLINT, INT, BIGINT: fixed-width little-endian signed integers;
    • BOOLEAN: one byte, 0 or 1;
    • DATE: four-byte little-endian days since epoch;
    • DECIMAL(p, s): an eight-byte little-endian unscaled value for p <= 18, and the minimal-length big-endian two's-complement unscaled value for p > 18;
    • STRING: UTF-8 bytes; BINARY: raw bytes.
  • Null handling: Distinguish null maps (outer length -1), empty maps (entry count 0), null values (BLOB length -1), and empty values (BLOB length 0). Reject null keys and duplicate keys, as Java does (apache/paimon#9107).
  • Value sources: Support both inline raw bytes and BlobDescriptor values. Descriptor values are materialized from their referenced storage when written, as Java does and as the ARRAY<BLOB> writer does.
  • Placeholder: Support data-evolution partial updates of MAP<K, BLOB> by recognizing an internal placeholder row on the write channel, aligned with Java BlobMapPlaceholder, and persisting it as a -2 entry. The fallback reader from #278 then resolves these rows from older sequence layers.
  • Write path: Route MAP<K, BLOB> fields to dedicated blob files in the data-evolution write path, and remove the write-side rejection in BlobUtils::ValidateContainerBlobWriteSchema.
  • Documentation: Update data_types.rst and write.rst, which currently state that Paimon C++ can read but not write MAP<kt, BLOB> columns.
  • Tests:
    • add unit tests for payload encoding of every supported key type, null/empty cases, null/duplicate key rejection, and error handling;
    • add round-trip tests with the C++ reader from #278, including placeholder fallback across sequence layers;
    • add Java/Python cross-compatibility tests, with C++ writing and Java/Python reading.
Anything else?
  • Depends on #283 / #278 for the reader side and builds on the ARRAY<BLOB> writer from #389 / #392.
  • TIME keys (apache/paimon#9009), compaction of data-evolution tables, paimon.map.selected-keys for MAP<K, BLOB> reads, and MAP<K, BLOB> in primary-key managed BLOB storage (#204) are out of scope and can be tracked separately.
  • No public API changes are expected. The storage format follows the existing Java spec.
Are you willing to submit a PR?
  • I'm willing to submit a PR!
Lingua principale
C++
Stelle
65
Fork
31
Merge medio
1g 23h
PR unite (30g)
64

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/paimon-cpp

Tutte le issue di apache/paimon-cpp

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.