feature: measure the payoff — agent pass rate with vs without the pack, graded by the existing headless harness
還沒有人認領這個 Issue。
評估
- 難度
- 5/5
- 預估耗時
- 一週以上
- 新手友好度
- 25/100
- Issue 類型
- 功能
- 描述清晰度
- 描述清楚
- 活躍度
- 活躍
- 技術堆疊
- blender, python
研究方向
Start by reading the existing headless grader at tests/smoke/run_example.py and the issue’s listed Blender API examples and CI versions. Then define reproducible evaluation tasks, run them with and without the plugin across versions and trials, and publish results on the landing page and README. Done means the reported pass-rate comparison includes task count, model, and date and can be reproduced from the repo.
由索引模型根據 Issue 內容生成。
描述
Value
This would be the project's biggest differentiator. Nothing in the README, site or ROADMAP measures whether the pack changes agent output. The landing page claims skills cover "the mistakes AI assistants actually make", with no data. A published number turns a content pack into evidence, and answers the skeptical visitor's first question.
Why this repo is unusually well placed
It already has what most eval efforts lack:
- deterministic headless graders (
tests/smoke/run_example.py, exit-code contracts, falsifiers); - 4.5, 5.1 and 5.2 Blender CI;
- a catalog of known-hard API contracts (slotted actions, temp_override, GN interface, VSE
.strips, export axes).
Claude Code also ships claude plugin eval (plugin eval suites).
Suggested approach
- Write about 20 task prompts with graders. Examples:
- "keyframe this object's rotation using 5.2 slotted actions"
- "export to Unreal-ready glTF"
- "bevel these edges with bmesh and free it"
- "add a COLOR strip to the VSE on 5.2"
- Each run: the agent writes a script, it runs headless against a checker, and you record pass or fail.
- Run with and without the plugin, per Blender version, for a named model, N trials each.
- Publish a table on the landing page and README, and refresh it on the weekly cron (or manually, for cost).
- Use the results to prioritise skill improvements, e.g. the skill trigger-description issue.
Done when
The landing page shows "agent pass rate X% → Y% with the pack (N tasks, model, date)", and it is reproducible from the repo.
- 主要語言
- Python
- 星號
- 17
- 分支
- 3
- 平均合併
- 23 小時 48 分鐘
- 30 天內合併 PR
- 3
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
TMHSDigital/Blender-Developer-Tools 的其他 Issue
-
bug ci priority: high
難度 2/5 1-3 小時 新手友好度 55/100
-
content-integrity priority: low showcase stale
難度 5/5 一週以上 新手友好度 45/100
TMHSDigital/Blender-Developer-Tools#499 · 1 則留言 ·
-
community documentation priority: high security stale
難度 3/5 1-2 天 新手友好度 56/100
TMHSDigital/Blender-Developer-Tools#392 · 2 則留言 ·
-
distribution documentation priority: high stale
難度 4/5 1-2 天 新手友好度 62/100
TMHSDigital/Blender-Developer-Tools#348 · 1 則留言 ·
查看 TMHSDigital/Blender-Developer-Tools 的全部 Issue
相似的 Issue
-
hive size:simple
難度 2/5 1-3 小時 新手友好度 78/100
維護者通常 1 天內回覆
-
A new VOLTTRON_HOME starts with agent isolation mode on (string comparison of a boolean default)未關閉
難度 2/5 1-3 小時 新手友好度 68/100
-
難度 2/5 1-3 小時 新手友好度 70/100
painted-porch/marcus#977 ·
維護者通常 1 天內回覆
-
llm translation
難度 2/5 1-3 小時 新手友好度 82/100
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 78/100
FAIRmat-NFDI/sand#239 ·
維護者通常 1 天內回覆