[Feature Request]: [Iceberg/Managed] Continuous unbounded streaming support in beam.managed.Write(ICEBERG) (Append & CDC)

Aperta
  • Qualcosa non va?
#41,226 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
java, python
Ambito
data-engineering, stream-processing

Direzione di ricerca

Start by reading the implementation of beam.managed.Write for Iceberg and the existing managed write tests; the issue does not name specific files. Trace how bounded writes build and commit Iceberg snapshots, then check streaming runner checkpoint and watermark integration. Done means unbounded writes support append and CDC modes with periodic or checkpoint-aligned commits and watermark metadata.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Description

Currently, beam.managed.Write(beam.managed.ICEBERG) is designed primarily for bounded (batch) PCollections. When writing unbounded streaming data (e.g., Kafka event logs, Pub/Sub streams, or continuous CDC streams from Debezium/Spanner), Beam lacks native streaming sink support to coordinate periodic snapshot commits against Iceberg catalogs (Google Cloud BigLake REST Catalog, Apache Polaris, AWS Glue).

Because Iceberg query engines (BigQuery, Trino, Spark) only read committed table snapshots, continuous streaming pipelines need native checkpoint-aligned snapshot commits so data becomes queryable with low latency.

Proposed Architecture

Extend beam.managed.Write(beam.managed.ICEBERG) to natively accept unbounded PCollections and manage streaming snapshot commits:

beam.managed.Write(
    beam.managed.ICEBERG,
    config={
        "table": "my_catalog.db.orders",
        "warehouse": "gs://my-bucket/lakehouse/wh",
        "commit_interval": "60s",      # maximum time between snapshot commits
        "mode": "append",              # "append" for event logs, or "merge-on-read" for CDC
        "catalog_properties": {
            "type": "rest",
            "uri": "https://biglake.googleapis.com/iceberg/v1/restcatalog",
        }
    }
)
Key Capabilities
  1. Unbounded Streaming Ingestion: Native execution on streaming runners (Dataflow Streaming Engine with Windmill, FlinkRunner).
  2. Checkpoint-Aligned Snapshot Commits: Flushes data files and writes atomic Iceberg snapshots coordinated with runner checkpoints or periodic commit intervals.
  3. Dual Ingestion Semantics:
    • Append-only Streams: High-throughput Parquet DataFile appends (Kafka, Pub/Sub).
    • CDC Streams (Merge-on-Read): Automatically generates Iceberg v2 Equality Delete files alongside DataFiles for row-level UPSERT and DELETE operations.
  4. Watermark Propagation: Records the pipeline's event-time watermark directly into the Iceberg snapshot summary metadata (snapshot.summary['beam.watermark']), giving downstream readers a verified consistency bound.
Lingua principale
Java
Stelle
8.7k
Fork
4.8k
Merge medio
2g 12h
PR unite (30g)
371

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/beam

Tutte le issue di apache/beam

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.