Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Feature] Support writing MAP<K, BLOB> fields

Đang mở
  • Có gì không ổn?
#415 0 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@SteNicholas đang làm issue này rồi.

Từ ngày 2/10/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

enhancement
Search before asking
  • I searched in the issues and found nothing similar.
Motivation

Java Paimon (apache/paimon#8635, with key types extended in apache/paimon#8963 and apache/paimon#9057) and PyPaimon can write top-level MAP<K, BLOB> columns to dedicated .blob files. #283 (PR #278) adds read-only MAP<K, BLOB> support to Paimon C++, and #389 (PR #392) adds write support for ARRAY<BLOB>, but C++ table creation (TableSchema::Create) and writes (FileStoreWrite::Create) still reject MAP<K, BLOB> fields through BlobUtils::ValidateContainerBlobWriteSchema. As a result, C++ users cannot write keyed collections of multimodal objects, such as the images of one sample keyed by camera name, to a Paimon table.

Solution

Add Java-compatible write support for top-level MAP<K, BLOB> fields declared with blob-field:

  • Schema validation: Keep the rules consistent with Java and with ARRAY<BLOB> (#392):
    • allow MAP<K, BLOB> only as a top-level field declared with blob-field;
    • reject it for blob-descriptor-field, blob-view-field, partition keys, and tables without data evolution;
    • reject unsupported key types and BLOB values nested more deeply, such as MAP<K, ARRAY<BLOB>> or ARRAY<MAP<K, BLOB>>.
  • Blob format writer: Extend BlobFormatWriter to encode each MAP<K, BLOB> row as the nested payload defined in the BLOB file spec and decoded by the C++ reader from #278. The payload contains:
    • map magic 1296188226;
    • version 1;
    • entry count;
    • concatenated key data;
    • concatenated BLOB data;
    • Delta-Varint key length index;
    • Delta-Varint BLOB length index;
    • key index length and BLOB index length.
  • Key encoding: Match the Java key encodings for the key types supported by the C++ reader:
    • TINYINT, SMALLINT, INT, BIGINT: fixed-width little-endian signed integers;
    • BOOLEAN: one byte, 0 or 1;
    • DATE: four-byte little-endian days since epoch;
    • DECIMAL(p, s): an eight-byte little-endian unscaled value for p <= 18, and the minimal-length big-endian two's-complement unscaled value for p > 18;
    • STRING: UTF-8 bytes; BINARY: raw bytes.
  • Null handling: Distinguish null maps (outer length -1), empty maps (entry count 0), null values (BLOB length -1), and empty values (BLOB length 0). Reject duplicate keys, as Java does (apache/paimon#9107). Also reject null keys, which Arrow map keys cannot hold; this differs from Java, which allows one null key per map.
  • Value sources: Support both inline raw bytes and BlobDescriptor values. Descriptor values are materialized from their referenced storage when written, as Java does and as the ARRAY<BLOB> writer does.
  • Placeholder: Support data-evolution partial updates of MAP<K, BLOB> by recognizing an internal placeholder row on the write channel, aligned with Java BlobMapPlaceholder, and persisting it as a -2 entry. The fallback reader from #278 then resolves these rows from older sequence layers.
  • Write path: Route MAP<K, BLOB> fields to dedicated blob files in the data-evolution write path, and remove the write-side rejection in BlobUtils::ValidateContainerBlobWriteSchema.
  • Documentation: Update data_types.rst and write.rst, which currently state that Paimon C++ can read but not write MAP<kt, BLOB> columns.
  • Tests:
    • add unit tests for payload encoding of every supported key type, null/empty cases, null/duplicate key rejection, and error handling;
    • add round-trip tests with the C++ reader from #278, including placeholder fallback across sequence layers;
    • add Java/Python cross-compatibility tests, with C++ writing and Java/Python reading.
Anything else?
  • Depends on #283 / #278 for the reader side and builds on the ARRAY<BLOB> writer from #389 / #392.
  • TIME keys (apache/paimon#9009), compaction of data-evolution tables, paimon.map.selected-keys for MAP<K, BLOB> reads, and MAP<K, BLOB> in primary-key managed BLOB storage (#204) are out of scope and can be tracked separately.
  • No public API changes are expected. The storage format follows the existing Java spec.
Are you willing to submit a PR?
  • I'm willing to submit a PR!
Ngôn ngữ chính
C++
Star
65
Fork
31
Merge trung bình
1 ngày 23 giờ
Pull request đã merge (30 ngày)
64

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/paimon-cpp

Tất cả issue của apache/paimon-cpp

Issue tương tự

Thêm issue về C++

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.