[Feature Request]: [Lakehouse/Python] Partition-aware zero-copy metadata translation from Delta Lake to Apache Iceberg

未关闭
  • 哪里不对？
#41,227 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@user-Param 已经在做这个了。

开始于 2026年11月12日。

  • #41234 来自 @user-Param —— 未关闭

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
活跃
技术栈
google-cloud, python
领域
cloud, data-engineering, databases, distributed-systems

调研方向

Start by reading the apache_beam.io.lakehouse package and how it handles existing table formats. Then trace how Delta transaction log JSON describes partitioned Parquet files and how Iceberg v2 manifests represent them. Done means a callable translation registers an Iceberg snapshot using existing data files, with partition mapping and catalog registration as described.

由索引模型根据 Issue 内容生成。

描述

Description

Migrating or synchronizing partitioned Delta Lake tables into Apache Iceberg currently requires full physical data rewriting. For multi-terabyte partitioned tables, this takes hours of cluster compute and substantial network/storage costs.
Because both formats use Parquet for data storage, an in-place zero-copy metadata translation can synthesize Iceberg manifests directly from Delta transaction logs.

Proposed Architecture

Add a partition-aware zero-copy metadata bridge in apache_beam.io.lakehouse:

from apache_beam.io.lakehouse import translate_delta_to_iceberg

# Synthesizes Iceberg v2 metadata from Delta partitioned transaction logs
metadata_path = translate_delta_to_iceberg(
    delta_table_uri="gs://my-bucket/delta/orders/",
    iceberg_warehouse="gs://my-bucket/iceberg/wh/",
    table_identifier="analytics.orders"
)
Key Capabilities
  1. Partition Specification Mapping: Parses Delta _delta_log/*.json partition definitions (e.g., date=2026-11-11/region=US/) into matching Iceberg PartitionSpec fields.
  2. Avro Manifest Generation: Generates Iceberg ManifestFile records pointing to existing Parquet files without copying data.
  3. Direct REST Catalog Registration: Automatically registers the synthesized snapshot in Google Cloud BigLake REST Catalog or Polaris in <5 seconds.
主要语言
Java
星标
8.7k
派生
4.8k
平均合并
2 天 12 小时
30 天内合并 PR
371

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/beam 的其他 Issue

查看 apache/beam 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。