[Feature] Roadmap for Paimon C++ 0.4.0

從連結的聚焦 issue 開始，尤其是關於即時寫入和 union 讀取的 #158、關於 changelog 產生的 #174、關於 Format Table 的 #170，以及關於 Data Evolution 的 #169 或 #204。在適用的情況下閱讀所參照的 PIP 文件，並選擇一個範圍明確的 roadmap 項目。完成的要求是進行一項聚焦且可供 review 的變更，並包含相關的 unit 和 integration 測試、文件以及相容性驗證。

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Looking ahead, Paimon C++ will focus on several areas:

• Storage and retrieval optimizations for vertical workloads, including efficient MAP storage, richer data types such as VECTOR, and complete index support.
• Making newly written data queryable in real time through pluggable real-time writes and memory/disk union reads.
• Following up on important capabilities from the broader Paimon community, including changelog production, Format Table, and extensions to Data Evolution.

Solution

1. Storage and retrieval optimizations for vertical workloads

• Support shared-shredding columnar storage for MAP<STRING, T> based on PIP-43, including adaptive physical columns, field mappings, overflow storage, and end-to-end reads and writes.
• Support richer data types, starting with VECTOR<T, N> based on PIP-40, including schema representation, storage, reads, writes, and Data Evolution.
• Complete File Index support by adding index generation to the existing read path.
• Continue improving storage layout, predicate pushdown, column pruning, point lookups, and index-assisted retrieval for workload-specific scenarios.

2. Making real-time data queryable

• Support pluggable real-time writes and memory/disk union reads based on PIP-46, tracked by #158.
• Make data held by an active writer queryable before it is committed into a snapshot.
• Guarantee consistent query results without missing or duplicate rows while writing, committing, and reclaiming memory segments concurrently.
• Support append tables first, followed by primary-key tables, deletion-vector mode, and additional production scenarios.

3. Following up on Paimon community capabilities

• Support changelog production, including input, lookup, and full-compaction modes, tracked by #174.
• Support Format Table reads and writes for Hive-style file directories, tracked by #170.
• Extend Data Evolution to work with deletion vectors, tracked by #169.
• Extend Data Evolution to primary-key tables and support compaction across evolved field groups, tracked by #204.
• Continue following relevant Paimon features and make them available through native Paimon C++ APIs where appropriate.

Each roadmap item should be delivered through focused issues and reviewable pull requests, with unit tests, integration tests, documentation, and compatibility validation where applicable.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!