[Feature Request]: [Python/YAML] Add native Expect / data quality PTransform with tagged DLQ outputs

未关闭
  • 哪里不对？
#41,223 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
活跃
技术栈
python, yaml
领域
data-engineering

调研方向

Start by reading the Python transform patterns in apache_beam.transforms and the Beam YAML transform registration and configuration paths. Then check how schema-aware transforms, tagged outputs, and standard Beam metrics are implemented. The issue describes the required rules and violation actions; done means the native transform and YAML registration support them with the requested main and DLQ outputs and metrics.

由索引模型根据 Issue 内容生成。

描述

Description

In production data pipelines (e.g. Bronze -> Silver -> Gold), validating records against declarative data quality rules is a fundamental requirement.
Currently in Apache Beam, implementing validation requires writing custom multi-output DoFns and branching logic. In Beam YAML, there is no declarative way to specify data quality rules with automated quarantine actions.

Proposed Architecture

Add an Expect Schema-Aware PTransform in apache_beam.transforms.expect (and register type: Expect in Beam YAML):

pipeline:
  transforms:
    - type: ReadFromCsv
      config: {path: 'gs://bucket/raw/*.csv'}
    - type: Expect
      config:
        rules:
          - name: positive_amount
            expression: 'amount > 0.0'
            on_violation: dlq
          - name: non_null_customer
            expression: 'customer_id IS NOT NULL'
            on_violation: fail
          - name: valid_email
            expression: "email LIKE '%@%'"
            on_violation: drop
        dlq_output: InvalidRecords
    - type: WriteToParquet
      input: Expect
      config: {path: 'gs://bucket/silver/'}
    - type: WriteToJson
      input: Expect.InvalidRecords
      config: {path: 'gs://bucket/dlq/'}
Key Capabilities
  1. Tagged Outputs: Emits Tag("main") for passed records and Tag("dlq") for quarantined records.
  2. Standard Beam Metrics: Emits beam:metric:user:expectation_passed:{rule} and expectation_quarantined:{rule} for real-time monitoring on Dataflow/Flink.
  3. Pluggable Violation Actions: dlq, drop, or fail (aborts worker).
主要语言
Java
星标
8.7k
派生
4.8k
平均合并
2 天 12 小时
30 天内合并 PR
371

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/beam 的其他 Issue

查看 apache/beam 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。