Author eval suite for agent `azure-principal-architect`
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 68/100
Research direction
Start by reading .github/agents/azure-principal-architect.agent.md and the existing suites under .github/evals/agents/, then use /agent-bench azure-principal-architect to draft the files. Run waza run .github/evals/agents/azure-principal-architect/eval.yaml -v and review the expanded entry in .github/evals/manifest.yaml. Done means the mock run is clean, positive and negative acceptance checks pass, and the PR includes the real-model run summary.
Written by the indexing model from the issue text.
Description
Agent
azure-principal-architect — source: .github/agents/azure-principal-architect.agent.md
Scope
Author the eval suite at .github/evals/agents/azure-principal-architect/:
-
eval.yaml— suite config (executor, model, graders) - At least 2 positive tasks under
tasks/positive-*.yaml - At least 1 negative task under
tasks/negative-*.yaml - Entry added to
.github/evals/manifest.yamlattier: expanded
Procedure
/agent-bench azure-principal-architectdrafts the suite from the live.agent.md.waza run .github/evals/agents/azure-principal-architect/eval.yaml -vlocally./agent-improve azure-principal-architectto iterate on graders.- Open PR.
- Mock CI runs automatically. A maintainer will dispatch a real-model run before merge.
Acceptance
- Suite runs cleanly in
mockexecutor. - At least one positive task passes in a real-model run.
- All negative tasks produce a refusal or out-of-scope acknowledgement.
-
manifest.yamlentry added; PR description includes the real-model run summary.
Conventions to follow
- Persona lock: refusal graders should accept the agent's own scope language.
- Prompt graders need
continue_session: truein their grader config.
Related
- Umbrella: #93
- Harness: #61
- Dominant language
- JavaScript
- Stars
- 269
- Forks
- 45
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 15
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Azure/git-ape
-
daily-status report
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
agentic-workflows
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
agentic-workflows
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
agentic-workflows
Difficulty 3/5 1-2 days Newbie friendliness 48/100
-
agentic-workflows
Difficulty 3/5 1-2 days Newbie friendliness 48/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
HarperFast/skills#96 ·
-
[Block] Latest Posts [Type] Bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Automattic/studio#4908 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
sugarlabs/musicblocks#8847 ·