Support Iceberg table encryption (Java-compatible)

This is a multi-PR design epic, so start by reading the vendored Arrow file encryption_internal_nossl.cc, which shows why Parquet encryption is compiled out, and the encryption-keys handling in table metadata. Newcomers should not take the epic as a whole; the phase 1 items are already assigned to the author and tracked separately (#989, #990, #991). Done means one of those scoped PRs merges with Java interop vectors passing.

Feature Description

What the feature achieves:
Adds Iceberg table encryption to iceberg-cpp, wire-compatible with the Java implementation (StandardEncryptionManager). iceberg-cpp can read tables that Java encrypted, and Java can read encrypted tables that iceberg-cpp writes. Data files, delete files, deletion vectors, manifests and manifest lists are encrypted; metadata.json is not, as in Java.

Design: follow the Java scheme exactly

Java defines the formats, so the C++ design matches it rather than adding anything new:

• Key hierarchy (envelope encryption). The table master key stays in the KMS (encryption.key-id). It wraps key encryption keys (KEKs), which are kept in the table metadata encryption-keys with a KEY_TIMESTAMP and rotated every 730 days. A KEK AES-GCM-encrypts the key metadata of each manifest list (AAD = KEK timestamp); the snapshot key-id points to that entry. Manifest DEKs live in the manifest list key_metadata, and data/delete file DEKs live in the manifest key_metadata, both in plaintext inside an encrypted file.
• Key metadata. StandardKeyMetadata: version byte 1 followed by an Avro record {encryption_key, aad_prefix, file_length}. The stored length is recorded for every encrypted file, including Parquet, because Java requires it for AGS1 reads (iceberg-rust found this in apache/iceberg-rust#3236).
• File formats. Parquet data and delete files use Parquet modular encryption (encrypted footer, PARE, footer key = DEK, AAD prefix not stored in the file). Manifests, manifest lists, Avro data files and Puffin deletion vectors use AES GCM Streams (AGS1, spec).
• Trust model. Encryption requires format v3. The table key ID cannot change once set. Catalogs must take the key ID from a trusted source, not only from metadata.json; the REST catalog creates its KMS client from client-side properties only.

Library layering: the core library iceberg stays free of crypto dependencies. It defines the interfaces (KeyManagementClient, EncryptionManager, EncryptingFileIO) and reaches the implementations through registries, the same pattern as ReaderFactoryRegistry and FileIORegistry. The bundle library iceberg_bundle (Arrow, Avro, Parquet) adds OpenSSL and the standard implementation. Cloud KMS clients are an optional library, iceberg_kms.

Why this feature is needed:
Problem: encrypted Iceberg tables written by Java (Spark, Flink, Trino with the Java library) cannot be used from C++ engines today.

Current gaps in iceberg-cpp:

• Table metadata encryption-keys is parsed, but there is no snapshot key-id, so encrypted snapshots cannot be resolved
• No crypto: no AES-GCM, no AGS1 streams, no key metadata, no KMS integration
• The vendored Arrow builds Parquet with encryption compiled out (encryption_internal_nossl.cc)
• Writers have no guard: writing to an encrypted table would silently write plaintext files
• Table-level encryption is listed as "not yet planned" in the v3 roadmap (#959)

Other implementations: Java has the core in main (Hive catalog wired end to end); iceberg-rust has merged an 18-step PR series (apache/iceberg-rust#2034); Python and Go have started.

User Experience

How users will use this feature:

Build:

# Bundle library with encryption (ON by default; requires OpenSSL)
-DICEBERG_ENCRYPTION=ON
# Optional: AWS KMS, Google Cloud KMS and Azure Key Vault clients
-DICEBERG_BUILD_KMS=ON

Configuration (same property names as Java):

Level	Property	Description
Catalog	encryption.kms-type	Built-in KMS client: aws, gcp or azure
Catalog	encryption.kms-impl	Name of a KMS client registered with KmsRegistry::Register
Table	encryption.key-id	ID of the table master key in the KMS; enables encryption
Table	encryption.data-key-length	Length of data keys: 16 (default), 24 or 32 bytes

Usage example:

// Built-in client from the (client-side) catalog properties, e.g. for REST:
//   encryption.kms-type = aws
//   client.region       = us-west-2
auto catalog = iceberg::rest::RestCatalog::Make(config);

// Or a custom KMS client
class MyKms : public iceberg::KeyManagementClient { /* WrapKey / UnwrapKey */ };
iceberg::KmsRegistry::Register("my-kms", [] { return std::make_unique<MyKms>(); });

// Tables loaded from the catalog get an EncryptingFileIO; scans, writers and
// commits encrypt and decrypt transparently. No API changes for readers/writers.

Read/write behavior:

• Scans of encrypted tables decrypt manifest lists, manifests, data files, delete files and deletion vectors
• Commits write encrypted files and add the new KEK and manifest list key entries to encryption-keys
• A catalog without a KMS client can load an encrypted table but fails cleanly on read or commit
• A wrong key fails with an error, not an exception or garbage data

Known Limitations (not supported)

These follow the Java implementation, which does not support them yet either:

• Column-level encryption (per-column keys) — apache/iceberg#14768
• Encrypted table and partition statistics files — apache/iceberg#17533
• Removing unused keys when snapshots expire — apache/iceberg#16352
• Re-wrapping keys with a new master key — apache/iceberg#18240
• REST KMS credential vending — apache/iceberg#17155
• Hive catalog: table operations are not implemented in iceberg-cpp yet, so there is nothing to wire
• Cloud credential providers: the KMS clients take explicit credentials or tokens; instance profiles, Google application default credentials and Azure managed identities are not supported (register a custom client instead)

API Changes:

New public APIs:

// Snapshot
std::optional<std::string> Snapshot::key_id;  // JSON "key-id"

// Key management
class KeyManagementClient {
  virtual Result<std::vector<uint8_t>> WrapKey(std::span<const uint8_t> key,
                                               std::string_view wrapping_key_id) = 0;
  virtual Result<std::vector<uint8_t>> UnwrapKey(std::span<const uint8_t> wrapped_key,
                                                 std::string_view wrapping_key_id) = 0;
};
class KmsRegistry;               // Register / Load / FromCatalogProperties

// Encryption
class EncryptionManager;         // StandardEncryptionManager in the bundle library
class EncryptionManagerRegistry;
class EncryptingFileIO;          // FileIO wrapper used by tables
struct StandardKeyMetadata;

// Reader/writer options
ReaderOptions::key_metadata;
WriterOptions::key_metadata;

Spec and Compatibility

• Does this change public interfaces/APIs? Yes — new classes above, Snapshot::key_id, ReaderOptions/WriterOptions::key_metadata
• Does this change storage format? No — it implements the existing v3 spec fields (encryption-keys, snapshot key-id, key_metadata) and the AGS1 spec, byte-compatible with Java
• Interoperability is verified both ways with Java:
  • C++ reads a Java-written encrypted table (AGS1 manifest lists and manifests, PARE Parquet, AGS1 Avro data files, an encrypted deletion vector)
  • Java reads a C++-written encrypted table and decrypts its deletion vector
• Small Java-generated fixtures (key metadata, AGS1 vectors, a wrapped KEK, a small encrypted table) are checked in as test resources

Task Breakdown

Proposed as a series of small PRs, following the order of the iceberg-rust series. Steps that do not depend on each other can be reviewed in parallel.

Phase 1: Foundations (no behavior change)

• 1. Build: Parquet modular encryption and OpenSSL in the vendored Arrow (plus zstd) — #989 (owner: @shangxinli)
• 2. Snapshot::key_id and JSON key-id — #990 (owner: @shangxinli)
• 3. StandardKeyMetadata with Java test vectors — #991 (owner: @shangxinli)
• 4. AES-GCM cipher and AGS1 input/output files (owner: @shangxinli)
• 5. KeyManagementClient, KmsRegistry, InMemoryKms (tests only) (owner: @shangxinli)
• 6. EncryptionManager and StandardEncryptionManager (KEK rotation, 1-hour unwrap cache) (owner: @shangxinli)
• 7. EncryptingFileIO (owner: @shangxinli)

Phase 2: Read path

• 8. Read encrypted manifest lists (owner: @shangxinli)
• 9. Read encrypted manifests (owner: @shangxinli)
• 10. Read encrypted Parquet/Avro data and delete files (owner: @shangxinli)

Phase 3: Write path

• 11. Block writes to encrypted tables until supported (no silent plaintext) (owner: @shangxinli)
• 12. KMS from catalog properties; in-memory, SQL and REST catalog wiring with a trusted key ID (owner: @shangxinli)
• 13. Write encrypted manifests and manifest lists (stored length through the Avro writer) (owner: @shangxinli)
• 14. Write encrypted data and delete files (owner: @shangxinli)
• 15. Encrypted Puffin deletion vectors (owner: @shangxinli)
• 16. Enable encrypted writes, end-to-end commit test, docs; also fixes manifest partition-spec metadata to be the fields array, which Java requires (owner: @shangxinli)

Phase 4: KMS clients

• 17. AWS KMS (SigV4), Google Cloud KMS and Azure Key Vault clients in optional iceberg_kms (owner: @shangxinli)

Feedback welcome on the library layering, the registry approach and the PR split.

Related Issues

• #959 — Track remaining Iceberg v3 feature support (table-level encryption metadata)
• apache/iceberg-rust#2034 — Support client-side encryption (iceberg-rust epic)
• apache/iceberg-rust#3236 — File length in key metadata for Java interop
• apache/iceberg#13225 — REST catalog encryption support (Java)
• apache/iceberg#16527 — Key metadata format in the spec
• apache/iceberg#16157 — Reading encrypted deletion vectors (Java)