Add an agent skill for datafusion-python extension authors
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 35/100
- Issue 类型
- 文档
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
调研方向
从 skills/datafusion_python/SKILL.md、.ai/skills/audit-skill-md/、docs/source/contributor-guide/ffi.md、两个示例 crate 以及公开的 with_extensions API 开始。弄清下游 agents 如何接收 skill,并等待 codec-id 缺口落地,然后根据文档记录的陷阱和当前示例定义 author skill 和 audit skill。完成的标准是:该指南拥有一条已达成共识的交付路径,以及一种可重复执行的漂移检查。
由索引模型根据 Issue 内容生成。
描述
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Writing a datafusion-python extension library means getting a handful of non-obvious lifetime and identity rules right, and there is currently no guidance an agent working in a downstream repo can find. The rules are spread across docs/source/contributor-guide/ffi.md, the docstrings on SessionContext.with_extensions / with_logical_extension_codec, .ai/skills/ffi-capsule-protocol/SKILL.md, and the two example crates. Each of those is written for someone already inside this repository.
The people who need it are not. Ballista and datafusion-distributed each build a separate cdylib plus Python glue against this project's FFI protocol, and they hit the same set of traps that #1679 was opened to fix: components bound to an intermediate context that is later collected, TaskContextProvider went out of scope over FFI boundary, codec ids that do not survive a process boundary, and write_parquet failing across the FFI codec because it has no file-format hooks. Every one of those is knowable in advance and none of them is discoverable from where those authors are standing.
Raised by @ntjohnson1 in review of #1679: "I wonder if it makes sense to have a todo for some datafusion python extension skill/s ... I do suspect it will require some guidance to make sure they are doing it safely."
Describe the solution you'd like
An agent skill for extension authors, plus a contributor-side audit skill that keeps it from rotting.
Location. A new skills/datafusion_python_extension/SKILL.md, sibling to the existing skills/datafusion_python/SKILL.md. Not .ai/skills/ — that tier is for agents working on this repository, and this audience is working on a different one. .ai/skills/ffi-capsule-protocol/SKILL.md stays where it is as the review trigger for changes to the capsule getters themselves, and the two cross-reference rather than duplicate.
Open question this issue needs to settle: how does a downstream agent get the file? skills/ is not currently included in the wheel, so an agent in a repo that depends on datafusion cannot see it. Options, roughly in order of how well they match "keeping it up to date shouldn't be too bad":
- Ship
skills/as package data in the wheel. Downstream gets the current version with the dependency they already have, and it tracks the release they are actually building against. - Publish to agentskills.io, the standard
CLAUDE.mdalready points at, sofind-skillsdiscovers it. - Document that downstream repos should vendor a copy. Cheapest to do, fastest to go stale.
Content. Drawn from the traps that #1672 and its three split PRs actually surfaced, not invented advice:
- The
__datafusion_session_extension__bundle protocol: create fresh components on every call, never cache bound components, never retain the context that was passed in. - Take the task-context provider off the supplied context. Never construct a
SessionContextinside an extension library. (Linkffi-capsule-protocol; do not restate it.) - Codec identity: the class-derived id,
__datafusion_codec_id__for pinning one, andanon:<uuid>for a barePyCapsule. A plan that has to decode in another process requires a declared id — this is the rule a distributed engine hits first. - Ownership: the context returned from
with_extensionsis the strong owner. ADataFramedoes not keep it alive.TaskContextProvider went out of scope over FFI boundaryis the symptom; holding the context is the cure. - Prefer
with_extensionsto hand-chainingwith_logical_extension_codec/with_physical_extension_codec/set_query_planner, and say why. - Known gap:
FFI_LogicalExtensionCodechas no vtable entries for file formats, sodf.write_csv/write_parquet/write_jsonfail across the boundary. Compose the FFI codec with aDefaultLogicalExtensionCodecfallback ontry_encode_file_format/try_decode_file_format.
Sequencing. This should land after the codec-id gap in with_extensions is resolved. SessionExtensionComponents currently has no way to declare an id for a codec handed over as a bare capsule, and with_extensions takes no codec_id=, so the only workaround is a Python shim that declares __datafusion_codec_id__. Writing the skill now would document that workaround as the intended design and require a rewrite when the API gains a way to express it.
Keeping it current. An .ai/skills/audit-extension-skill/ contributor skill following the existing .ai/skills/audit-skill-md/ pattern: diff the skill's claims against the two example crates and the current public API, and report drift. This is the part worth automating, because the failure mode is the skill silently describing a protocol that has since moved.
Describe alternatives you've considered
Extend .ai/skills/ffi-capsule-protocol instead of adding a skill. Wrong audience and wrong trigger. That skill fires when someone touches a __datafusion_*__ getter in this repository. An extension author is not touching those; they are implementing against them, and they need a build-it-from-scratch narrative rather than a review checklist.
Extend docs/source/contributor-guide/ffi.md instead. Already the reference material, and #1679 adds substantially to it. But it is organized around the protocol rather than around the task, and it lives at a URL a downstream agent has no reason to fetch. The skill should point at ffi.md for depth rather than replace it.
Assert in CI that the skill can regenerate the three-library example. This was the original suggestion and it is the right instinct — the example is the best available proof the guidance is sufficient — but it does not survive contact with a workflow. Generated output varies between runs, so a byte-comparison fails on noise and a fuzzy comparison asserts nothing. The deterministic half is the audit skill above. The full generate-from-skill-and-run-the-tests exercise is real value in the wrong shape for CI; it belongs on the release checklist as a manual step.
Additional context
Reference implementations the skill should point at, both exercised in CI via .github/workflows/test.yml:
examples/datafusion-ffi-example/— table providers, UDFs, catalog providers, optimizer rules, extension codecs.examples/datafusion-ffi-query-planner-example/— query planner, andMyPlannerExtensioninsrc/extension.rsas a complete Rust implementation of__datafusion_session_extension__.
Follows up on #1679 (part 3 of the #1672 split).
- 主要语言
- Python
- 星标
- 605
- 派生
- 176
- 平均合并
- 1 天 23 小时
- 30 天内合并 PR
- 8
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
apache/datafusion-python 的其他 Issue
-
enhancement
难度 2/5 1-3 小时 新手友好度 70/100
apache/datafusion-python#1757 ·
-
documentation
难度 2/5 1-3 小时 新手友好度 72/100
apache/datafusion-python#1726 ·
-
难度 2/5 半天 新手友好度 88/100
apache/datafusion-python#1691 ·
-
bug
难度 2/5 1-3 小时 新手友好度 78/100
apache/datafusion-python#1644 ·
-
enhancement
难度 5/5 一周以上 新手友好度 30/100
apache/datafusion-python#1737 ·
查看 apache/datafusion-python 的全部 Issue
相似的 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 90/100
learningequality/ricecooker#747 ·
-
难度 2/5 1-3 小时 新手友好度 68/100
BSData/horus-heresy-3rd-edition#3171 ·
-
enhancement
难度 2/5 1-3 小时 新手友好度 72/100
-
难度 2/5 1-3 小时 新手友好度 76/100
run-llama/llama_index#23199 ·
-
难度 2/5 1-3 小时 新手友好度 84/100
KhronosGroup/glTF-Blender-IO#2769 ·