Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Feature] Support writing MAP<K, BLOB> fields

Abierto
  • ¿Algo no está bien?
#415 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@SteNicholas ya está trabajando en esto.

Desde el 2/10/2026.

  • #416 de @SteNicholas — abierto

Evaluación

Este issue todavía no se ha evaluado.

Descripción

enhancement
Search before asking
  • I searched in the issues and found nothing similar.
Motivation

Java Paimon (apache/paimon#8635, with key types extended in apache/paimon#8963 and apache/paimon#9057) and PyPaimon can write top-level MAP<K, BLOB> columns to dedicated .blob files. #283 (PR #278) adds read-only MAP<K, BLOB> support to Paimon C++, and #389 (PR #392) adds write support for ARRAY<BLOB>, but C++ table creation (TableSchema::Create) and writes (FileStoreWrite::Create) still reject MAP<K, BLOB> fields through BlobUtils::ValidateContainerBlobWriteSchema. As a result, C++ users cannot write keyed collections of multimodal objects, such as the images of one sample keyed by camera name, to a Paimon table.

Solution

Add Java-compatible write support for top-level MAP<K, BLOB> fields declared with blob-field:

  • Schema validation: Keep the rules consistent with Java and with ARRAY<BLOB> (#392):
    • allow MAP<K, BLOB> only as a top-level field declared with blob-field;
    • reject it for blob-descriptor-field, blob-view-field, partition keys, and tables without data evolution;
    • reject unsupported key types and BLOB values nested more deeply, such as MAP<K, ARRAY<BLOB>> or ARRAY<MAP<K, BLOB>>.
  • Blob format writer: Extend BlobFormatWriter to encode each MAP<K, BLOB> row as the nested payload defined in the BLOB file spec and decoded by the C++ reader from #278. The payload contains:
    • map magic 1296188226;
    • version 1;
    • entry count;
    • concatenated key data;
    • concatenated BLOB data;
    • Delta-Varint key length index;
    • Delta-Varint BLOB length index;
    • key index length and BLOB index length.
  • Key encoding: Match the Java key encodings for the key types supported by the C++ reader:
    • TINYINT, SMALLINT, INT, BIGINT: fixed-width little-endian signed integers;
    • BOOLEAN: one byte, 0 or 1;
    • DATE: four-byte little-endian days since epoch;
    • DECIMAL(p, s): an eight-byte little-endian unscaled value for p <= 18, and the minimal-length big-endian two's-complement unscaled value for p > 18;
    • STRING: UTF-8 bytes; BINARY: raw bytes.
  • Null handling: Distinguish null maps (outer length -1), empty maps (entry count 0), null values (BLOB length -1), and empty values (BLOB length 0). Reject duplicate keys, as Java does (apache/paimon#9107). Also reject null keys, which Arrow map keys cannot hold; this differs from Java, which allows one null key per map.
  • Value sources: Support both inline raw bytes and BlobDescriptor values. Descriptor values are materialized from their referenced storage when written, as Java does and as the ARRAY<BLOB> writer does.
  • Placeholder: Support data-evolution partial updates of MAP<K, BLOB> by recognizing an internal placeholder row on the write channel, aligned with Java BlobMapPlaceholder, and persisting it as a -2 entry. The fallback reader from #278 then resolves these rows from older sequence layers.
  • Write path: Route MAP<K, BLOB> fields to dedicated blob files in the data-evolution write path, and remove the write-side rejection in BlobUtils::ValidateContainerBlobWriteSchema.
  • Documentation: Update data_types.rst and write.rst, which currently state that Paimon C++ can read but not write MAP<kt, BLOB> columns.
  • Tests:
    • add unit tests for payload encoding of every supported key type, null/empty cases, null/duplicate key rejection, and error handling;
    • add round-trip tests with the C++ reader from #278, including placeholder fallback across sequence layers;
    • add Java/Python cross-compatibility tests, with C++ writing and Java/Python reading.
Anything else?
  • Depends on #283 / #278 for the reader side and builds on the ARRAY<BLOB> writer from #389 / #392.
  • TIME keys (apache/paimon#9009), compaction of data-evolution tables, paimon.map.selected-keys for MAP<K, BLOB> reads, and MAP<K, BLOB> in primary-key managed BLOB storage (#204) are out of scope and can be tracked separately.
  • No public API changes are expected. The storage format follows the existing Java spec.
Are you willing to submit a PR?
  • I'm willing to submit a PR!
Lenguaje dominante
C++
Estrellas
65
Forks
31
Merge medio
1 d 14 h
PR fusionados (30 d)
60

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/paimon-cpp

Todos los issues de apache/paimon-cpp

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.