Author eval suite for skill `azure-cost-estimator`

Open
#99 0 comments 1 reaction 2 assignees View on GitHub

@sendtoshailesh is already working on this.

Since May 26, 2026.

  • #168 by @copilot-swe-agent — open

Assessment

This issue has not been assessed yet.

Description

AI-evals enhancement

Skill

azure-cost-estimator — source: .github/skills/azure-cost-estimator/SKILL.md

Scope

Author the eval suite at .github/evals/azure-cost-estimator/:

  • eval.yaml — suite config (executor, model, graders)
  • At least 2 positive tasks under tasks/positive-*.yaml
  • At least 1 negative task under tasks/negative-*.yaml
  • Entry added to .github/evals/manifest.yaml at tier: expanded

Procedure

  1. /skill-bench azure-cost-estimator drafts the suite from the live SKILL.md.
  2. waza run .github/evals/azure-cost-estimator/eval.yaml -v locally.
  3. /skill-improve azure-cost-estimator to iterate on graders.
  4. Open PR.
  5. Mock CI runs automatically. A maintainer will dispatch a real-model run before merge.

Acceptance

  • Suite runs cleanly in mock executor.
  • At least one positive task passes in a real-model run.
  • All negative tasks produce a refusal or out-of-scope acknowledgement.
  • manifest.yaml entry added; PR description includes the real-model run summary.

Conventions to follow

  • Persona lock: refusal graders should accept the agent's own scope language.
  • Don't add required_skills to a skill_invocation grader unless the skill genuinely invokes those sub-skills.
  • Prompt graders need continue_session: true in their grader config.

Related

  • Umbrella: #93
  • Harness: #61
Dominant language
JavaScript
Stars
269
Forks
45
Avg merge
1d 4h
Merged PRs (30d)
15

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Azure/git-ape

All issues in Azure/git-ape

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.