flyteorg/flyte

[Core feature] Make execution_id and execution_domain available for plugins

Open

#6,308 opened on 2025年3月5日

GitHub で見る
 (3 comments) (0 reactions) (2 assignees)Python (3,705 stars) (378 forks)batch import
enhancementgood first issuehelp wanted

説明

Motivation: Why do you think this is important?

BACKGROUND

https://flyte-org.slack.com/archives/CP2HDHKE1/p1741181647724639

ISSUE

We are trying to fetch Flyte execution domain in a Spark task running in Databricks using the Fyte Databricks plugin.

We tried to use the same method that works well in Python tasks running on K8s (flytekit.current_context().execution_id.domain)

Unfortunately, this approach does not work in case of Databricks plugin. It would be nice to pass this info in the API so that we can access it in tasks.

Goal: What should the final outcome look like, ideally?

We are able to fetch execution domain in tasks.

Describe alternatives you've considered

Pass the domain as workflow input param.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

  • Yes

Have you read the Code of Conduct?

  • Yes

コントリビューターガイド

技術スタック
python
領域
backenddata
Issue 種別
feature
難度新しい貢献者にとっての実装難度です。1 は非常に小さな変更、5 は専門的な作業です。
4
推定時間経験ある貢献者が調査、実装、テスト、pull request 準備にかけるおおよその時間範囲です。
1-2 days
活動状況Issue が今どれだけ取り組みやすいかを示します。新しい、活発、古い、ブロック中、またはメンテナー入力待ちなどです。
active
明確さ期待される変更、受け入れ条件、次の手順がどれだけ明確かを示します。
mostly clear
前提条件
Understanding of Flyte plugin systemPython
初心者向け度初回貢献者にどれだけ取り組みやすいかを 1-100 で推定したスコアです。
25
調査方針
The issue requires making execution id and execution domain accessible in Flyte plugins, specifically the Databricks plugin. Currently, Python tasks on K8s use flytekit.current context().execution id.domain, but this doesn't work in Databricks. The research should investigate how context is propagated in the plugin SDK and how to extend it to support execution metadata. Look at the plugin interface in flytekit and the Databricks plugin code to understand the current limitations and propose a solution to pass execution information to task executions.