feature: measure the payoff — agent pass rate with vs without the pack, graded by the existing headless harness

Aperta
#373 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
blender, python

Direzione di ricerca

Start by reading the existing headless grader at tests/smoke/run_example.py and the issue’s listed Blender API examples and CI versions. Then define reproducible evaluation tasks, run them with and without the plugin across versions and trials, and publish results on the landing page and README. Done means the reported pass-rate comparison includes task count, model, and date and can be reproduced from the repo.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

feature marketing priority: high stale testing

Value

This would be the project's biggest differentiator. Nothing in the README, site or ROADMAP measures whether the pack changes agent output. The landing page claims skills cover "the mistakes AI assistants actually make", with no data. A published number turns a content pack into evidence, and answers the skeptical visitor's first question.

Why this repo is unusually well placed

It already has what most eval efforts lack:

  • deterministic headless graders (tests/smoke/run_example.py, exit-code contracts, falsifiers);
  • 4.5, 5.1 and 5.2 Blender CI;
  • a catalog of known-hard API contracts (slotted actions, temp_override, GN interface, VSE .strips, export axes).

Claude Code also ships claude plugin eval (plugin eval suites).

Suggested approach

  1. Write about 20 task prompts with graders. Examples:
    • "keyframe this object's rotation using 5.2 slotted actions"
    • "export to Unreal-ready glTF"
    • "bevel these edges with bmesh and free it"
    • "add a COLOR strip to the VSE on 5.2"
  2. Each run: the agent writes a script, it runs headless against a checker, and you record pass or fail.
  3. Run with and without the plugin, per Blender version, for a named model, N trials each.
  4. Publish a table on the landing page and README, and refresh it on the weekly cron (or manually, for cost).
  5. Use the results to prioritise skill improvements, e.g. the skill trigger-description issue.

Done when

The landing page shows "agent pass rate X% → Y% with the pack (N tasks, model, date)", and it is reproducible from the repo.

Lingua principale
Python
Stelle
17
Fork
3
Merge medio
23h 48m
PR unite (30g)
3

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di TMHSDigital/Blender-Developer-Tools

Tutte le issue di TMHSDigital/Blender-Developer-Tools

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.