Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

BUG SDK JUnit results are never published because the runner OS condition is wrong

Đã đóng
  • Có gì không ổn?
#2,776 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 2 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
68/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
github-actions
Lĩnh vực
ci-cd, testing

Hướng nghiên cứu

Start with .github/workflows/build_and_test.yml, job main-job, and the Publish Pytest Results step; compare the matrix OS labels with the runner.os values and review the existing pinned publisher action. Validate success, test-failure, cancellation, and missing-XML paths, ensuring raw JUnit artifacts remain downloadable, test failures stay authoritative, and fork/security boundaries are preserved.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug help wanted
Describe the bug

The Publish Pytest Results step in .github/workflows/build_and_test.yml uses:

if: runner.os == 'ubuntu-latest'

ubuntu-latest is a runner label from matrix.os; the corresponding runner.os value is Linux. The condition therefore never matches. In the audited September 21-22 main-branch window, all 504 SDK result-publisher steps were skipped, across successful, failed, and cancelled matrix jobs. This does not mean the tests themselves were all skipped.

The workflow also has no independent upload of raw SDK JUnit XML. This makes real failures harder to inspect, including the MOSSBench setup error in job 106560764233, whose workflow aggregate is cancelled even though the test job failed.

A second boundary matters when correcting the OS comparison: a plain step condition without an explicit status check normally inherits success-only gating. We need reports from failed test steps too, not just successful runs.

Steps/Code to Reproduce
  1. Inspect .github/workflows/build_and_test.yml, job main-job, step Publish Pytest Results.
  2. Compare the matrix.os values (ubuntu-latest, windows-latest, macos-latest) with the runner.os context (Linux, Windows, macOS).
  3. Inspect main run 35668881112, attempt 1. The unit-test job writes junit/test-results.xml, but its publisher step is skipped and no independent SDK JUnit artifact is exposed.

The condition itself is a deterministic reproduction; no live target, new CI run, or probabilistic test failure is needed to establish the mismatch.

Expected Results

Publish SDK test results for the intended supported runners whenever test output exists, including after unit-test failure, and retain downloadable raw JUnit evidence.

Acceptance criteria:

  • Compare the correct context/value while preserving the intended supported-runner policy, for example Linux publication using runner.os == 'Linux' rather than a runner-label string.
  • Add explicit outcome gating so failed tests do not suppress their result publication. Handle cancellation or missing XML honestly rather than manufacturing a successful test result.
  • Retain raw JUnit XML through an independent artifact-upload path, including failed jobs where output exists, with matrix-safe artifact names.
  • Keep the unit-test exit status authoritative. Reporting must not hide the failure or make the test step continue on error.
  • Preserve fork/security boundaries and least-privilege permissions; do not solve publication by executing untrusted PR code in a privileged event.
  • Validate the success, test-failure, and missing-output paths without changing test selection, assertions, thresholds, or retry policy.

This is a workflow reporting repair, not a fix for the underlying MOSSBench or Azure E2E failures. Use existing pinned actions and repository conventions.

Actual Results

Fixed audit window: September 21, 2026, 13:11:11 UTC through September 22, 2026, 13:11:11 UTC.

  • 21 build_and_test main runs; 504 SDK matrix jobs.
  • Matrix-job outcomes: 422 success, 1 failure, 81 cancelled.
  • Publish Pytest Results: skipped in all 504 jobs. Some cancelled jobs never reach it, but the OS condition also prevents it in completed Linux jobs.
  • No independent SDK JUnit artifact upload in the workflow.
  • The failed MOSSBench job mentions generated junit/test-results.xml in its log, but the report is not available as a standalone artifact.
Screenshots

N/A. The workflow source and linked run/job steps provide the evidence.

Versions
  • Workflow: .github/workflows/build_and_test.yml, main-job.
  • Matrix: Ubuntu/Windows/macOS, Python 3.11-3.14, default and all-extras installations.
  • Observed main snapshot: 2016c4a8566bd66253d431ff38400bade4c77fa3; the same condition remains on main at filing.
  • Publisher: EnricoMi/publish-unit-test-result-action, pinned to d0a4676d0e0b938bc201470d88276b7c74c712b3 (v2.24.0).
  • Browser and local package-version snapshot: N/A for this GitHub Actions expression defect.

Related observed failure: #2775 tracks the MOSSBench setup error. This issue repairs reporting and must not hide or change that test failure.

Ngôn ngữ chính
Python
Star
4.6k
Fork
944
Merge trung bình
3 ngày 1 giờ
Pull request đã merge (30 ngày)
278

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

  • Không có Dockerfile hay tệp Docker Compose
  • Có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của microsoft/PyRIT

Tất cả issue của microsoft/PyRIT

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.