Author eval suite for agent `git-ape`
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 58/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- github, yaml
- Domain
- ai-infra-agents, testing
Research direction
Read .github/agents/git-ape.agent.md and compare the established sub-agent suites once their baselines are stable. Run /agent-bench git-ape, then validate .github/evals/agents/git-ape/eval.yaml with waza run ... -v; done means the mock suite is clean, real-model and negative-task acceptance checks pass, and .github/evals/manifest.yaml includes the expanded-tier entry.
Written by the indexing model from the issue text.
Description
Agent
git-ape — source: .github/agents/git-ape.agent.md
Scope
Author the eval suite at .github/evals/agents/git-ape/:
-
eval.yaml— suite config (executor, model, graders) - At least 2 positive tasks under
tasks/positive-*.yaml - At least 1 negative task under
tasks/negative-*.yaml - Entry added to
.github/evals/manifest.yamlattier: expanded
Dependency note
git-ape is the orchestrator agent. Defer this suite until most sub-agent suites (azure-requirements-gatherer, azure-template-generator, azure-resource-deployer, azure-iac-exporter) are stable — regressions in this suite are easier to root-cause when each sub-agent has its own established baseline.
Procedure
/agent-bench git-apedrafts the suite from the live.agent.md.waza run .github/evals/agents/git-ape/eval.yaml -vlocally./agent-improve git-apeto iterate on graders.- Open PR.
- Mock CI runs automatically. A maintainer will dispatch a real-model run before merge.
Acceptance
- Suite runs cleanly in
mockexecutor. - At least one positive task passes in a real-model run.
- All negative tasks produce a refusal or out-of-scope acknowledgement.
-
manifest.yamlentry added; PR description includes the real-model run summary.
Conventions to follow
- Persona lock: refusal graders should accept the agent's own scope language.
- Prompt graders need
continue_session: truein their grader config. - This agent has destructive tools through delegation. Apply the same "no real deploy" rule as
azure-resource-deployer: positive tasks grade safety-contract behavior, not real Azure execution.
Related
- Umbrella: #93
- Harness: #61
- Dominant language
- JavaScript
- Stars
- 269
- Forks
- 45
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 15
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Azure/git-ape
-
daily-status report
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
agentic-workflows
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
agentic-workflows
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
agentic-workflows
Difficulty 3/5 1-2 days Newbie friendliness 48/100
-
agentic-workflows
Difficulty 3/5 1-2 days Newbie friendliness 48/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
HarperFast/skills#96 ·
-
[Block] Latest Posts [Type] Bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Automattic/studio#4908 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
sugarlabs/musicblocks#8847 ·