Extend SessionContext.with_extensions to cover additional extension points (functions, catalogs, object stores)
維護者通常 9 天內回覆
還沒有人認領這個 Issue。
評估
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 新手友好度
- 52/100
- Issue 類型
- 功能
- 描述清晰度
- 基本清楚
- 活躍度
- 冷清
研究方向
從 crates/core/src/context.rs 中的 SessionExtensionComponents、SessionExtensionExportable 和 with_extensions 交易開始;將它們與 examples/datafusion-ffi-query-planner-example 中的 MyPlannerExtension 範例進行比較。先從建議的 udfs 和 catalog_providers 範圍開始,然後解決衝突處理和註冊順序問題。當 bundles 能夠透過 with_extensions 以原子方式安裝這些元件,且不破壞現有 bundles 時,即表示完成。
由索引模型根據 Issue 內容生成。
描述
Is your feature request related to a problem or challenge?
#1672 adds SessionContext.with_extensions(*extensions), which installs extension bundles atomically: each bundle implements __datafusion_session_extension__(ctx) and returns a SessionExtensionComponents containing logical extension codecs, physical extension codecs, and optionally a query planner. Those components carry weak task-context providers, so the atomic protocol exists to guarantee they bind to the exact context returned to the user.
Extension libraries typically provide more than codecs and a planner. A distributed engine or data-source library may also ship scalar/aggregate/window/table functions, catalog providers, table providers, object stores, and physical optimizer rules. Today the user must install those with separate register_* calls after with_extensions, which defeats the goal of a one-line installation of everything a library provides.
Describe the solution you'd like
Extend SessionExtensionComponents with additional optional fields so a single bundle can declare every extension point its library offers:
udfs,udafs,udwfs,udtfs(or a singlefunctionstuple with type dispatch) — accepting both FFI-capsule and Python-native definitions, flowing through the existingregister_*pathstable_providers: tuple[tuple[str, provider], ...]— named table registrationscatalog_providers: tuple[tuple[str, provider], ...]object_stores: tuple[tuple[str, store], ...]— keyed by schemephysical_optimizer_rules
SessionExtensionComponents is a frozen dataclass with defaulted fields and the Rust _install_extensions helper is private, so this is backward compatible and can be added incrementally. A distributed engine bundle could then install codecs, planner, its UDFs, and a scheduler-backed catalog in one call:
ctx = SessionContext(config).with_extensions(MyEngineExtension(address))
Declarative components are preferred over having factories call ctx.register_udf(...) on the context view directly: the host can validate every component before mutating anything, the configuration-only contract for factories stays honest, and installation order becomes explicit (codecs, then planner, then registrations, with fallible steps first).
Design decisions to settle during implementation:
- Catalog list sharing. The derived context shares its catalog provider list with the source context, so catalog registrations are visible in the source and are not rolled back if a later step fails. Registering catalogs last, after all fallible steps, closes most of the hole. Cloning the catalog provider list at derivation would give true isolation but diverges from the semantics of every other
with_*method. - Name collisions. Two bundles registering the same UDF name or catalog name should probably be an error (matching the one-planner rule) rather than last-wins, since registrations have no fall-through semantics like codec chains do.
Suggested starting scope: udfs and catalog_providers, with collision handling as an error.
Describe alternatives you've considered
Factories can already register functions and catalogs imperatively on the context view passed to __datafusion_session_extension__, since the view shares the destination context. This works but hides side effects inside the factory, cannot be validated up front, and leaves partial registrations behind on failure.
Keeping registrations as separate user-facing register_* calls after with_extensions remains possible, but each library then documents its own multi-step setup recipe, which is the situation with_extensions was introduced to remove.
Additional context
Follow-up to #1672. Relevant pieces there: SessionExtensionComponents, SessionExtensionExportable, the with_extensions transaction in crates/core/src/context.rs, and the MyPlannerExtension example bundle in examples/datafusion-ffi-query-planner-example.
- 主要語言
- Python
- 星號
- 605
- 分支
- 176
- 平均合併
- 1 天 23 小時
- 30 天內合併 PR
- 9
環境準備
我們還沒有檢查這個專案的環境設定檔。先看它的 README,通用步驟見我們的新手貢獻指南。
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
apache/datafusion-python 的其他 Issue
-
bug
難度 2/5 1-3 小時 新手友好度 75/100
apache/datafusion-python#1765 ·
維護者通常 9 天內回覆
-
難度 2/5 1-3 小時 新手友好度 78/100
apache/datafusion-python#1760 ·
維護者通常 9 天內回覆
-
enhancement
難度 2/5 1-3 小時 新手友好度 70/100
apache/datafusion-python#1757 ·
維護者通常 9 天內回覆
-
documentation
難度 2/5 1-3 小時 新手友好度 72/100
apache/datafusion-python#1726 ·
維護者通常 9 天內回覆
-
難度 2/5 半天 新手友好度 88/100
apache/datafusion-python#1691 ·
維護者通常 9 天內回覆
查看 apache/datafusion-python 的全部 Issue
相似的 Issue
-
needs triage
難度 2/5 1-3 小時 新手友好度 78/100
維護者通常 2 天內回覆
-
難度 2/5 1-3 小時 新手友好度 82/100
openvinotoolkit/openvino_notebooks#3665 ·
維護者通常 1 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 86/100
維護者通常 1 天內回覆
-
docs
難度 2/5 1-3 小時 新手友好度 88/100
維護者通常 1 天內回覆
-
benchmark-gap
難度 2/5 1-3 小時 新手友好度 78/100
維護者通常 1 天內回覆