feathr-ai/feathr

[FR] Feathr Lineage is missing UDF info

Open

#561 创建于 2022年8月9日

在 GitHub 查看
 (0 评论) (0 反应) (0 负责人)Scala (1,929 star) (244 fork)batch import
good first issue

描述

Willingness to contribute

No. I cannot contribute a bug fix at this time.

Feature Request Proposal

Feathr UI should show UDF information in the lineage graph. Currently it's not showed.

Motivation

What is the use case for this feature?

Showing the UDF info could be helpful for end users to know how the input is transformed.

Details

No response

What component(s) does this bug affect?

  • Python Feathr Client: This is the client users use to interact with most of our API. Mostly written in Python.
  • Computation Engine: The computation engine that execute the actual feature join and generation work. Mostly in Scala and Spark.
  • Feature Registry API Layer: The storage layer supports SQL, Purview(Atlas). The API layer is in Python(FAST API)
  • Feature Registry Web UI layer: The Web UI for feature registry. Written in React with a few UI frameworks.

贡献者指南

技术栈
reactpythonfastapi
领域
frontenddata
议题类型
feature
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
3
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
1-2 days
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
stale
清晰度议题是否清楚说明期望改动、验收标准和下一步。
mostly clear
前置要求
React basicsPython FastAPILineage data flowFeathr source code structure
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
40
研究方向
Start by examining the Feathr UI codebase, specifically the React components for lineage graph in the Feature Registry Web UI layer. Look for where the graph data is fetched and rendered, and trace back to the Python FastAPI endpoint that serves lineage data. UDF info might be missing from the API response or not captured in the graph model. Investigate the computation engine (Scala/Spark) to see if UDF metadata is recorded, and then modify the API and UI to include it.