Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Agent evals] Add Vally coverage for Python skills and onboard the shared eval pipeline

Abierto
  • ¿Algo no está bien?
#49,057 0 comentarios 1 reacción 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
45/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
azure, python
Área
ci-cd, devops, testing-qa

Línea de trabajo

Start with the listed Skill eval authoring workflow and inspect each Python skill's .github/skills//SKILL.md, then compare the eval.yaml and .vally.yaml examples. Review eng/pipelines/skill-eval.yml requirements and run strict Vally lint plus focused local evals using the mock MCP. Done means all nine skills have routed coverage, required boundaries and graders pass, and the report-only pipeline publishes its standard artifacts.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Context

This tracks the Python rollout for Azure/azure-sdk-tools#16348

Before enabling the shared language-repository eval pipeline, add current-format Vally coverage for the Python-specific Agent Skills. Shared azsdk-common-* skills already carry centrally synced evals and should not be duplicated or independently modified here.

Use these sources of truth:

Do not copy the eval-authoring-* meta-skills into this repository.

Skill eval work

Add Vally coverage for every Python-owned skill under .github/skills/:

  • agentic-workflows
  • create-api-review
  • create-package-skill
  • find-package-skill
  • fix-black
  • fix-mypy
  • fix-pylint
  • fix-sphinx
  • generate-api-markdown

For each skill:

  • Read its SKILL.md, especially its activation boundaries, guardrails, and invoked tools or commands.
  • Put routing and capability coverage in .github/skills/<skill>/evals/eval.yaml. Split into additional <behavior>.eval.yaml files only when coverage is genuinely large.
  • Add at least three realistic triggers and three anti-triggers using skill-invocation.
  • For a boundary prompt, mount the closest competing skill, require that skill, and disallow the skill under test.
  • Add capability or guardrail stimuli using tool-calls and deterministic output, file, or command graders where applicable.
  • Supply concrete Python package names, repository paths, diagnostics, target branches, or other identifiers needed to complete the request.
  • Use hermetic fixtures and the mock MCP. Do not use secrets, network-dependent writes, or production targets.
  • Set tags.area to the exact skill name, use defaults:, and set scoring.threshold: 0.8.

Cover meaningful boundaries, including:

  • fix-black, fix-mypy, fix-pylint, and fix-sphinx; each diagnostic must route only to the correct fixer.
  • create-api-review versus generate-api-markdown and shared APIView feedback resolution.
  • create-package-skill versus find-package-skill and ordinary package modification.
  • agentic-workflows versus Azure SDK CI, package, or release workflows.

Capability fixtures for the fixer skills should contain small representative diagnostics and source files. Grade the expected correction and verification command rather than relying on a response keyword. Do not contact GitHub or create real pull requests from evals.

Repository configuration

Add .github/skills/.vally.yaml following the tools-repo layout:

  • Register every local and synced skill eval directory.
  • Define azsdk-mcp-mock using ../../artifacts/mcp/mock/azsdk-mock.dll.
  • Set cwd: . for the MCP server so paths remain stable in Vally trial workspaces.
  • Add .github/skills/.gitignore for Vally results and caches.

Pipeline

Add eng/pipelines/skill-eval.yml as a thin entrypoint extending:

/eng/common/pipelines/templates/stages/archetype-eval.yml

Required configuration:

  • Declare Azure/azure-sdk-tools as repository resource azure-sdk-tools, using endpoint Azure and refs/heads/main.
  • Use /eng/common/pipelines/templates/steps/eval-mcp-setup.yml.
  • Set TestType: mock and toolsRepo: azure-sdk-tools.
  • Set vallyRoot: .github/skills.
  • Discover both */evals/eval.yaml and */evals/*.eval.yaml so local and synced suites run.
  • Use threshold: 0.8, failOnFailedTests: false, and shardTimeoutInMinutes: 20.
  • Reference /eng/pipelines/templates/variables/image.yml and AzSDK_Eval_Variable_group.
  • Start manual/scheduled on main with trigger: none and pr: none. Add selective PR triggers or blocking only after the pilot is stable.

Do not copy or customize shared eng/common framework files in this repository. Framework fixes belong in azure-sdk-tools and must arrive through the normal sync.

Register the YAML as the python - skill eval Azure DevOps pipeline and authorize its repository resource and variable group.

Acceptance criteria

  • Every current Python-owned skill has current-format Vally routing coverage.
  • The four fixer skills have discriminating boundary and capability tests.
  • Supported capabilities and critical guardrails have deterministic graders.
  • Shared azsdk-common-* evals remain centrally sourced.
  • Strict Vally lint passes for every added or changed eval.
  • Focused local eval runs pass and their commands and pass counts are recorded in the PR.
  • The scheduled, report-only pipeline completes and publishes standard summary, JUnit, and failure-debug artifacts.
  • No production writes, credentials, or real customer identifiers are used.
  • Any deferred live-only behavior is documented in a follow-up issue.
Lenguaje dominante
Python
Estrellas
5.6k
Forks
3.4k
Merge medio
2 d 7 h
PR fusionados (30 d)
206

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Azure/azure-sdk-for-python

Todos los issues de Azure/azure-sdk-for-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.