Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

[Feature] Support writing MAP<K, BLOB> fields

Aberta
  • Algo errado?
#415 0 comentários 0 reações 1 responsável Ver no GitHub

Mantenedores costumam responder em até 1 dia

@SteNicholas já está trabalhando nisso.

Desde 2/10/2026.

Avaliação

Esta issue ainda não foi avaliada.

Descrição

enhancement
Search before asking
  • I searched in the issues and found nothing similar.
Motivation

Java Paimon (apache/paimon#8635, with key types extended in apache/paimon#8963 and apache/paimon#9057) and PyPaimon can write top-level MAP<K, BLOB> columns to dedicated .blob files. #283 (PR #278) adds read-only MAP<K, BLOB> support to Paimon C++, and #389 (PR #392) adds write support for ARRAY<BLOB>, but C++ table creation (TableSchema::Create) and writes (FileStoreWrite::Create) still reject MAP<K, BLOB> fields through BlobUtils::ValidateContainerBlobWriteSchema. As a result, C++ users cannot write keyed collections of multimodal objects, such as the images of one sample keyed by camera name, to a Paimon table.

Solution

Add Java-compatible write support for top-level MAP<K, BLOB> fields declared with blob-field:

  • Schema validation: Keep the rules consistent with Java and with ARRAY<BLOB> (#392):
    • allow MAP<K, BLOB> only as a top-level field declared with blob-field;
    • reject it for blob-descriptor-field, blob-view-field, partition keys, and tables without data evolution;
    • reject unsupported key types and BLOB values nested more deeply, such as MAP<K, ARRAY<BLOB>> or ARRAY<MAP<K, BLOB>>.
  • Blob format writer: Extend BlobFormatWriter to encode each MAP<K, BLOB> row as the nested payload defined in the BLOB file spec and decoded by the C++ reader from #278. The payload contains:
    • map magic 1296188226;
    • version 1;
    • entry count;
    • concatenated key data;
    • concatenated BLOB data;
    • Delta-Varint key length index;
    • Delta-Varint BLOB length index;
    • key index length and BLOB index length.
  • Key encoding: Match the Java key encodings for the key types supported by the C++ reader:
    • TINYINT, SMALLINT, INT, BIGINT: fixed-width little-endian signed integers;
    • BOOLEAN: one byte, 0 or 1;
    • DATE: four-byte little-endian days since epoch;
    • DECIMAL(p, s): an eight-byte little-endian unscaled value for p <= 18, and the minimal-length big-endian two's-complement unscaled value for p > 18;
    • STRING: UTF-8 bytes; BINARY: raw bytes.
  • Null handling: Distinguish null maps (outer length -1), empty maps (entry count 0), null values (BLOB length -1), and empty values (BLOB length 0). Reject null keys and duplicate keys, as Java does (apache/paimon#9107).
  • Value sources: Support both inline raw bytes and BlobDescriptor values. Descriptor values are materialized from their referenced storage when written, as Java does and as the ARRAY<BLOB> writer does.
  • Placeholder: Support data-evolution partial updates of MAP<K, BLOB> by recognizing an internal placeholder row on the write channel, aligned with Java BlobMapPlaceholder, and persisting it as a -2 entry. The fallback reader from #278 then resolves these rows from older sequence layers.
  • Write path: Route MAP<K, BLOB> fields to dedicated blob files in the data-evolution write path, and remove the write-side rejection in BlobUtils::ValidateContainerBlobWriteSchema.
  • Documentation: Update data_types.rst and write.rst, which currently state that Paimon C++ can read but not write MAP<kt, BLOB> columns.
  • Tests:
    • add unit tests for payload encoding of every supported key type, null/empty cases, null/duplicate key rejection, and error handling;
    • add round-trip tests with the C++ reader from #278, including placeholder fallback across sequence layers;
    • add Java/Python cross-compatibility tests, with C++ writing and Java/Python reading.
Anything else?
  • Depends on #283 / #278 for the reader side and builds on the ARRAY<BLOB> writer from #389 / #392.
  • TIME keys (apache/paimon#9009), compaction of data-evolution tables, paimon.map.selected-keys for MAP<K, BLOB> reads, and MAP<K, BLOB> in primary-key managed BLOB storage (#204) are out of scope and can be tracked separately.
  • No public API changes are expected. The storage format follows the existing Java spec.
Are you willing to submit a PR?
  • I'm willing to submit a PR!
Linguagem predominante
C++
Estrelas
65
Forks
31
Merge médio
1d 23h
PRs com merge (30d)
64

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/paimon-cpp

Todas as issues de apache/paimon-cpp

Issues semelhantes

Mais issues de C++

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.