Author eval suite for agent `azure-requirements-gatherer`
@arnaudlh is already working on this.
Since May 21, 2026.
Assessment
This issue has not been assessed yet.
Description
Agent
azure-requirements-gatherer — source: .github/agents/azure-requirements-gatherer.agent.md
Scope
Author the eval suite at .github/evals/agents/azure-requirements-gatherer/:
-
eval.yaml— suite config (executor, model, graders) - At least 2 positive tasks under
tasks/positive-*.yaml - At least 1 negative task under
tasks/negative-*.yaml - Entry added to
.github/evals/manifest.yamlattier: expanded
Notes
This agent has an interactive Q&A loop that would deadlock under copilot-sdk. The eval MUST exercise the agent's documented headless mode — the task prompt should explicitly set that mode so the agent produces a single non-interactive response.
Procedure
/agent-bench azure-requirements-gathererdrafts the suite from the live.agent.md.waza run .github/evals/agents/azure-requirements-gatherer/eval.yaml -vlocally./agent-improve azure-requirements-gathererto iterate on graders.- Open PR.
- Mock CI runs automatically. A maintainer will dispatch a real-model run before merge.
Acceptance
- Suite runs cleanly in
mockexecutor. - At least one positive task passes in a real-model run.
- All negative tasks produce a refusal or out-of-scope acknowledgement.
-
manifest.yamlentry added; PR description includes the real-model run summary.
Conventions to follow
- Persona lock: refusal graders should accept the agent's own scope language.
- Prompt graders need
continue_session: truein their grader config.
Related
- Umbrella: #93
- Harness: #61
- Dominant language
- JavaScript
- Stars
- 269
- Forks
- 45
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 15
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Azure/git-ape
-
daily-status report
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
agentic-workflows
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
agentic-workflows
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
agentic-workflows
Difficulty 3/5 1-2 days Newbie friendliness 48/100
-
agentic-workflows
Difficulty 3/5 1-2 days Newbie friendliness 48/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
HarperFast/skills#96 ·
-
[Block] Latest Posts [Type] Bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Automattic/studio#4908 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
sugarlabs/musicblocks#8847 ·