[Agent evals] Add Vally coverage for Python skills and onboard the shared eval pipeline

Start with the listed Skill eval authoring workflow and inspect each Python skill's .github/skills//SKILL.md, then compare the eval.yaml and .vally.yaml examples. Review eng/pipelines/skill-eval.yml requirements and run strict Vally lint plus focused local evals using the mock MCP. Done means all nine skills have routed coverage, required boundaries and graders pass, and the report-only pipeline publishes its standard artifacts.

Context

This tracks the Python rollout for Azure/azure-sdk-tools#16348

Before enabling the shared language-repository eval pipeline, add current-format Vally coverage for the Python-specific Agent Skills. Shared azsdk-common-* skills already carry centrally synced evals and should not be duplicated or independently modified here.

Use these sources of truth:

• Skill eval authoring workflow
• Eval authoring guide
• Skill eval layout example
• Vally configuration example
• Shared skill-eval entrypoint
• Shared eval archetype

Do not copy the eval-authoring-* meta-skills into this repository.

Skill eval work

Add Vally coverage for every Python-owned skill under .github/skills/:

• agentic-workflows
• create-api-review
• create-package-skill
• find-package-skill
• fix-black
• fix-mypy
• fix-pylint
• fix-sphinx
• generate-api-markdown

For each skill:

• Read its SKILL.md, especially its activation boundaries, guardrails, and invoked tools or commands.
• Put routing and capability coverage in .github/skills/<skill>/evals/eval.yaml. Split into additional <behavior>.eval.yaml files only when coverage is genuinely large.
• Add at least three realistic triggers and three anti-triggers using skill-invocation.
• For a boundary prompt, mount the closest competing skill, require that skill, and disallow the skill under test.
• Add capability or guardrail stimuli using tool-calls and deterministic output, file, or command graders where applicable.
• Supply concrete Python package names, repository paths, diagnostics, target branches, or other identifiers needed to complete the request.
• Use hermetic fixtures and the mock MCP. Do not use secrets, network-dependent writes, or production targets.
• Set tags.area to the exact skill name, use defaults:, and set scoring.threshold: 0.8.

Cover meaningful boundaries, including:

• fix-black, fix-mypy, fix-pylint, and fix-sphinx; each diagnostic must route only to the correct fixer.
• create-api-review versus generate-api-markdown and shared APIView feedback resolution.
• create-package-skill versus find-package-skill and ordinary package modification.
• agentic-workflows versus Azure SDK CI, package, or release workflows.

Capability fixtures for the fixer skills should contain small representative diagnostics and source files. Grade the expected correction and verification command rather than relying on a response keyword. Do not contact GitHub or create real pull requests from evals.

Repository configuration

Add .github/skills/.vally.yaml following the tools-repo layout:

• Register every local and synced skill eval directory.
• Define azsdk-mcp-mock using ../../artifacts/mcp/mock/azsdk-mock.dll.
• Set cwd: . for the MCP server so paths remain stable in Vally trial workspaces.
• Add .github/skills/.gitignore for Vally results and caches.

Pipeline

Add eng/pipelines/skill-eval.yml as a thin entrypoint extending:

/eng/common/pipelines/templates/stages/archetype-eval.yml

Required configuration:

• Declare Azure/azure-sdk-tools as repository resource azure-sdk-tools, using endpoint Azure and refs/heads/main.
• Use /eng/common/pipelines/templates/steps/eval-mcp-setup.yml.
• Set TestType: mock and toolsRepo: azure-sdk-tools.
• Set vallyRoot: .github/skills.
• Discover both */evals/eval.yaml and */evals/*.eval.yaml so local and synced suites run.
• Use threshold: 0.8, failOnFailedTests: false, and shardTimeoutInMinutes: 20.
• Reference /eng/pipelines/templates/variables/image.yml and AzSDK_Eval_Variable_group.
• Start manual/scheduled on main with trigger: none and pr: none. Add selective PR triggers or blocking only after the pilot is stable.

Do not copy or customize shared eng/common framework files in this repository. Framework fixes belong in azure-sdk-tools and must arrive through the normal sync.

Register the YAML as the python - skill eval Azure DevOps pipeline and authorize its repository resource and variable group.

Acceptance criteria

• Every current Python-owned skill has current-format Vally routing coverage.
• The four fixer skills have discriminating boundary and capability tests.
• Supported capabilities and critical guardrails have deterministic graders.
• Shared azsdk-common-* evals remain centrally sourced.
• Strict Vally lint passes for every added or changed eval.
• Focused local eval runs pass and their commands and pass counts are recorded in the PR.
• The scheduled, report-only pipeline completes and publishes standard summary, JUnit, and failure-debug artifacts.
• No production writes, credentials, or real customer identifiers are used.
• Any deferred live-only behavior is documented in a follow-up issue.