feature: measure the payoff — agent pass rate with vs without the pack, graded by the existing headless harness

未关闭
#373 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
25/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
活跃
技术栈
blender, python

调研方向

Start by reading the existing headless grader at tests/smoke/run_example.py and the issue’s listed Blender API examples and CI versions. Then define reproducible evaluation tasks, run them with and without the plugin across versions and trials, and publish results on the landing page and README. Done means the reported pass-rate comparison includes task count, model, and date and can be reproduced from the repo.

由索引模型根据 Issue 内容生成。

描述

feature marketing priority: high stale testing

Value

This would be the project's biggest differentiator. Nothing in the README, site or ROADMAP measures whether the pack changes agent output. The landing page claims skills cover "the mistakes AI assistants actually make", with no data. A published number turns a content pack into evidence, and answers the skeptical visitor's first question.

Why this repo is unusually well placed

It already has what most eval efforts lack:

  • deterministic headless graders (tests/smoke/run_example.py, exit-code contracts, falsifiers);
  • 4.5, 5.1 and 5.2 Blender CI;
  • a catalog of known-hard API contracts (slotted actions, temp_override, GN interface, VSE .strips, export axes).

Claude Code also ships claude plugin eval (plugin eval suites).

Suggested approach

  1. Write about 20 task prompts with graders. Examples:
    • "keyframe this object's rotation using 5.2 slotted actions"
    • "export to Unreal-ready glTF"
    • "bevel these edges with bmesh and free it"
    • "add a COLOR strip to the VSE on 5.2"
  2. Each run: the agent writes a script, it runs headless against a checker, and you record pass or fail.
  3. Run with and without the plugin, per Blender version, for a named model, N trials each.
  4. Publish a table on the landing page and README, and refresh it on the weekly cron (or manually, for cost).
  5. Use the results to prioritise skill improvements, e.g. the skill trigger-description issue.

Done when

The landing page shows "agent pass rate X% → Y% with the pack (N tasks, model, date)", and it is reproducible from the repo.

主要语言
Python
星标
17
派生
3
平均合并
23 小时 48 分钟
30 天内合并 PR
3

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

TMHSDigital/Blender-Developer-Tools 的其他 Issue

查看 TMHSDigital/Blender-Developer-Tools 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。