apache/dolphinscheduler

[DSIP-79][Task] Add Datavines task to better support data quality

Open

#16,113 建立於 2024年6月3日

在 GitHub 查看
 (10 留言) (2 反應) (1 負責人)Java (4,324 fork)batch import
DSIPhelp wanted

倉庫指標

Star
 (11,659 star)
PR 合併指標
 (平均合併 2天 21小時) (30 天內合併 22 個 PR)

描述

Search before asking

  • I had searched in the DSIP and found no similar DSIP.

Motivation

DataVines is an easy-to-use data quality service platform that supports multiple metric. https://github.com/datavane/datavines

  • Datavines supports executing multiple metrics in one job.
  • Datavines supports execution status dashboard and data quality report.
  • Datavines supports plug-in extensions for components such as metric, data sources, error data storage, and execution engines.
  • Jdbc engines can be used to execute data quality tasks instead of solely relying on Spark engines.

Design Detail

Sript mode

  1. config data quality job in datavines image

  2. get the job config scipt file

  3. Add datavines job node in workflow, and configure the script image

API Mode

  1. config data quality job in datavines image

  2. get the jobId

  3. Add datavines job node in workflow, and configure the datavines api address and jobId

Compatibility, Deprecation, and Migration Plan

No response

Test Plan

No response

Code of Conduct

貢獻者指南