apache/dolphinscheduler
在 GitHub 查看[DSIP-79][Task] Add Datavines task to better support data quality
Open
#16,113 建立於 2024年6月3日
DSIPhelp wanted
倉庫指標
- Star
- (11,659 star)
- PR 合併指標
- (平均合併 2天 21小時) (30 天內合併 22 個 PR)
描述
Search before asking
- I had searched in the DSIP and found no similar DSIP.
Motivation
DataVines is an easy-to-use data quality service platform that supports multiple metric. https://github.com/datavane/datavines
- Datavines supports executing multiple metrics in one job.
- Datavines supports execution status dashboard and data quality report.
- Datavines supports plug-in extensions for components such as metric, data sources, error data storage, and execution engines.
- Jdbc engines can be used to execute data quality tasks instead of solely relying on Spark engines.
Design Detail
Sript mode
-
config data quality job in datavines
-
get the job config scipt file
-
Add datavines job node in workflow, and configure the script
API Mode
-
config data quality job in datavines
-
get the jobId
-
Add datavines job node in workflow, and configure the datavines api address and jobId
Compatibility, Deprecation, and Migration Plan
No response
Test Plan
No response
Code of Conduct
- I agree to follow this project's Code of Conduct