Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Agent evals] Add Vally coverage for Python skills and onboard the shared eval pipeline

Aperta
  • Qualcosa non va?
#49,057 0 commenti 1 reazione 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
45/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
azure, python
Ambito
ci-cd, devops, testing-qa

Direzione di ricerca

Start with the listed Skill eval authoring workflow and inspect each Python skill's .github/skills//SKILL.md, then compare the eval.yaml and .vally.yaml examples. Review eng/pipelines/skill-eval.yml requirements and run strict Vally lint plus focused local evals using the mock MCP. Done means all nine skills have routed coverage, required boundaries and graders pass, and the report-only pipeline publishes its standard artifacts.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Context

This tracks the Python rollout for Azure/azure-sdk-tools#16348

Before enabling the shared language-repository eval pipeline, add current-format Vally coverage for the Python-specific Agent Skills. Shared azsdk-common-* skills already carry centrally synced evals and should not be duplicated or independently modified here.

Use these sources of truth:

Do not copy the eval-authoring-* meta-skills into this repository.

Skill eval work

Add Vally coverage for every Python-owned skill under .github/skills/:

  • agentic-workflows
  • create-api-review
  • create-package-skill
  • find-package-skill
  • fix-black
  • fix-mypy
  • fix-pylint
  • fix-sphinx
  • generate-api-markdown

For each skill:

  • Read its SKILL.md, especially its activation boundaries, guardrails, and invoked tools or commands.
  • Put routing and capability coverage in .github/skills/<skill>/evals/eval.yaml. Split into additional <behavior>.eval.yaml files only when coverage is genuinely large.
  • Add at least three realistic triggers and three anti-triggers using skill-invocation.
  • For a boundary prompt, mount the closest competing skill, require that skill, and disallow the skill under test.
  • Add capability or guardrail stimuli using tool-calls and deterministic output, file, or command graders where applicable.
  • Supply concrete Python package names, repository paths, diagnostics, target branches, or other identifiers needed to complete the request.
  • Use hermetic fixtures and the mock MCP. Do not use secrets, network-dependent writes, or production targets.
  • Set tags.area to the exact skill name, use defaults:, and set scoring.threshold: 0.8.

Cover meaningful boundaries, including:

  • fix-black, fix-mypy, fix-pylint, and fix-sphinx; each diagnostic must route only to the correct fixer.
  • create-api-review versus generate-api-markdown and shared APIView feedback resolution.
  • create-package-skill versus find-package-skill and ordinary package modification.
  • agentic-workflows versus Azure SDK CI, package, or release workflows.

Capability fixtures for the fixer skills should contain small representative diagnostics and source files. Grade the expected correction and verification command rather than relying on a response keyword. Do not contact GitHub or create real pull requests from evals.

Repository configuration

Add .github/skills/.vally.yaml following the tools-repo layout:

  • Register every local and synced skill eval directory.
  • Define azsdk-mcp-mock using ../../artifacts/mcp/mock/azsdk-mock.dll.
  • Set cwd: . for the MCP server so paths remain stable in Vally trial workspaces.
  • Add .github/skills/.gitignore for Vally results and caches.

Pipeline

Add eng/pipelines/skill-eval.yml as a thin entrypoint extending:

/eng/common/pipelines/templates/stages/archetype-eval.yml

Required configuration:

  • Declare Azure/azure-sdk-tools as repository resource azure-sdk-tools, using endpoint Azure and refs/heads/main.
  • Use /eng/common/pipelines/templates/steps/eval-mcp-setup.yml.
  • Set TestType: mock and toolsRepo: azure-sdk-tools.
  • Set vallyRoot: .github/skills.
  • Discover both */evals/eval.yaml and */evals/*.eval.yaml so local and synced suites run.
  • Use threshold: 0.8, failOnFailedTests: false, and shardTimeoutInMinutes: 20.
  • Reference /eng/pipelines/templates/variables/image.yml and AzSDK_Eval_Variable_group.
  • Start manual/scheduled on main with trigger: none and pr: none. Add selective PR triggers or blocking only after the pilot is stable.

Do not copy or customize shared eng/common framework files in this repository. Framework fixes belong in azure-sdk-tools and must arrive through the normal sync.

Register the YAML as the python - skill eval Azure DevOps pipeline and authorize its repository resource and variable group.

Acceptance criteria

  • Every current Python-owned skill has current-format Vally routing coverage.
  • The four fixer skills have discriminating boundary and capability tests.
  • Supported capabilities and critical guardrails have deterministic graders.
  • Shared azsdk-common-* evals remain centrally sourced.
  • Strict Vally lint passes for every added or changed eval.
  • Focused local eval runs pass and their commands and pass counts are recorded in the PR.
  • The scheduled, report-only pipeline completes and publishes standard summary, JUnit, and failure-debug artifacts.
  • No production writes, credentials, or real customer identifiers are used.
  • Any deferred live-only behavior is documented in a follow-up issue.
Lingua principale
Python
Stelle
5.6k
Fork
3.4k
Merge medio
2g 12h
PR unite (30g)
210

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Azure/azure-sdk-for-python

Tutte le issue di Azure/azure-sdk-for-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.