Improve Databricks operators with query tags

(5 件のコメント) (0 件のリアクション) (0 人の担当者)Python (16,781 件のフォーク)batch import

good first issuekind:featureprovider:databricks

Repository metrics

Stars: (44,809 個のスター)
PR merge metrics: (平均マージ 7d 18h) (30d で 834 merged PRs)

説明

Description

Databricks supports query tags which allows us to track where the source of a query is on Databricks UI. Reference: https://docs.databricks.com/aws/en/sql/user/queries/query-tags

Currently when a query is sent to Databricks, on Databricks UI, the only info we can see is the query is from Airflow, but there is no other metadata like airflow dag id, task id, run id and etc.

Use case/motivation

With query tags instrumented, it will be easy for organizations to track and do Databricks cost contribution across different Airflow DAGs.

Related issues

No response

Are you willing to submit a PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

コントリビューターガイド

調査方針: Examine the existing Databricks operators in the Airflow providers package (e.g., airflow/providers/databricks/operators/). Understand the query tag documentation from Databricks to see how tags are passed. Consider adding a parameter to the operators for query tags and ensure they are attached to each query execution. Check for any existing PRs or discussions on the issue for implementation details.
技術スタック: python
領域: backenddata
Issue 種別: 機能
難度: 3
推定時間: 1-2日
活動状況: アクティブ
明確さ: 明確
前提条件: Basic understanding of Apache Airflow operatorsFamiliarity with Databricks SQLPython programming
初心者向け度: 50