Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Agent evals] Add Vally coverage for Python skills and onboard the shared eval pipeline

Đang mở
  • Có gì không ổn?
#49,057 0 bình luận 1 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
45/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
azure, python
Lĩnh vực
ci-cd, devops, testing-qa

Hướng nghiên cứu

Start with the listed Skill eval authoring workflow and inspect each Python skill's .github/skills//SKILL.md, then compare the eval.yaml and .vally.yaml examples. Review eng/pipelines/skill-eval.yml requirements and run strict Vally lint plus focused local evals using the mock MCP. Done means all nine skills have routed coverage, required boundaries and graders pass, and the report-only pipeline publishes its standard artifacts.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Context

This tracks the Python rollout for Azure/azure-sdk-tools#16348

Before enabling the shared language-repository eval pipeline, add current-format Vally coverage for the Python-specific Agent Skills. Shared azsdk-common-* skills already carry centrally synced evals and should not be duplicated or independently modified here.

Use these sources of truth:

Do not copy the eval-authoring-* meta-skills into this repository.

Skill eval work

Add Vally coverage for every Python-owned skill under .github/skills/:

  • agentic-workflows
  • create-api-review
  • create-package-skill
  • find-package-skill
  • fix-black
  • fix-mypy
  • fix-pylint
  • fix-sphinx
  • generate-api-markdown

For each skill:

  • Read its SKILL.md, especially its activation boundaries, guardrails, and invoked tools or commands.
  • Put routing and capability coverage in .github/skills/<skill>/evals/eval.yaml. Split into additional <behavior>.eval.yaml files only when coverage is genuinely large.
  • Add at least three realistic triggers and three anti-triggers using skill-invocation.
  • For a boundary prompt, mount the closest competing skill, require that skill, and disallow the skill under test.
  • Add capability or guardrail stimuli using tool-calls and deterministic output, file, or command graders where applicable.
  • Supply concrete Python package names, repository paths, diagnostics, target branches, or other identifiers needed to complete the request.
  • Use hermetic fixtures and the mock MCP. Do not use secrets, network-dependent writes, or production targets.
  • Set tags.area to the exact skill name, use defaults:, and set scoring.threshold: 0.8.

Cover meaningful boundaries, including:

  • fix-black, fix-mypy, fix-pylint, and fix-sphinx; each diagnostic must route only to the correct fixer.
  • create-api-review versus generate-api-markdown and shared APIView feedback resolution.
  • create-package-skill versus find-package-skill and ordinary package modification.
  • agentic-workflows versus Azure SDK CI, package, or release workflows.

Capability fixtures for the fixer skills should contain small representative diagnostics and source files. Grade the expected correction and verification command rather than relying on a response keyword. Do not contact GitHub or create real pull requests from evals.

Repository configuration

Add .github/skills/.vally.yaml following the tools-repo layout:

  • Register every local and synced skill eval directory.
  • Define azsdk-mcp-mock using ../../artifacts/mcp/mock/azsdk-mock.dll.
  • Set cwd: . for the MCP server so paths remain stable in Vally trial workspaces.
  • Add .github/skills/.gitignore for Vally results and caches.

Pipeline

Add eng/pipelines/skill-eval.yml as a thin entrypoint extending:

/eng/common/pipelines/templates/stages/archetype-eval.yml

Required configuration:

  • Declare Azure/azure-sdk-tools as repository resource azure-sdk-tools, using endpoint Azure and refs/heads/main.
  • Use /eng/common/pipelines/templates/steps/eval-mcp-setup.yml.
  • Set TestType: mock and toolsRepo: azure-sdk-tools.
  • Set vallyRoot: .github/skills.
  • Discover both */evals/eval.yaml and */evals/*.eval.yaml so local and synced suites run.
  • Use threshold: 0.8, failOnFailedTests: false, and shardTimeoutInMinutes: 20.
  • Reference /eng/pipelines/templates/variables/image.yml and AzSDK_Eval_Variable_group.
  • Start manual/scheduled on main with trigger: none and pr: none. Add selective PR triggers or blocking only after the pilot is stable.

Do not copy or customize shared eng/common framework files in this repository. Framework fixes belong in azure-sdk-tools and must arrive through the normal sync.

Register the YAML as the python - skill eval Azure DevOps pipeline and authorize its repository resource and variable group.

Acceptance criteria

  • Every current Python-owned skill has current-format Vally routing coverage.
  • The four fixer skills have discriminating boundary and capability tests.
  • Supported capabilities and critical guardrails have deterministic graders.
  • Shared azsdk-common-* evals remain centrally sourced.
  • Strict Vally lint passes for every added or changed eval.
  • Focused local eval runs pass and their commands and pass counts are recorded in the PR.
  • The scheduled, report-only pipeline completes and publishes standard summary, JUnit, and failure-debug artifacts.
  • No production writes, credentials, or real customer identifiers are used.
  • Any deferred live-only behavior is documented in a follow-up issue.
Ngôn ngữ chính
Python
Star
5.6k
Fork
3.4k
Merge trung bình
2 ngày 12 giờ
Pull request đã merge (30 ngày)
210

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của Azure/azure-sdk-for-python

Tất cả issue của Azure/azure-sdk-for-python

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.