Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Clarify identity semantics for parameterized tests and merged retries

Đang mở
  • Có gì không ổn?
#66 4 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
35/100
Loại issue
Tài liệu
Độ rõ ràng
Cần làm rõ
Mức độ hoạt động
Sôi nổi
Lĩnh vực
documentation, testing-qa

Hướng nghiên cứu

Start with the identity model described in the issue, then read PRs #57 and #62 and the referenced MTP files: CtrfReportEngine.JsonTestWriter.cs and UnitTestElementExtensions.cs. The work is done when the specification documents a normative model for parameterized tests and merged retries, including one worked example and guidance MTP can implement.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Producer context

Microsoft.Testing.Platform (MTP) produces CTRF for multiple test frameworks, so it cannot assume every framework assigns the same stability semantics to its native test UID.

Today the MTP CTRF reporter keeps the framework-provided UID under tests[].extra.uid rather than claiming it is a CTRF testId:

For MSTest specifically, that UID is a versioned hash of the assembly file name, fully-qualified test name, parameter types, and (for data-driven tests) the row index:

This is stable for many rebuilds, but a rename, assembly rename, or reordered dynamic data source can change it.

What the current spec resolves

The identity model is clear that:

  • testId identifies a logical test case and should be stable across runs within the producer's documented scope;
  • executionId is unique for one execution of that logical test within a run;
  • attemptId identifies an individual entry in retryAttempts;
  • identity fields are optional and opaque to consumers.

PR #57 and PR #62 provide useful general guidance, but we could not find guidance for parameterized cases or for reports transformed by retry merging.

Questions

  1. For a parameterized/data-driven method, is each argument row a distinct logical test case that should have its own testId, or should all rows share the method's testId and be distinguished by parameters plus executionId?
  2. If rows are distinct test cases, should a generated testId be based on canonical parameter values (stable across data-source reordering) rather than a row index? How should producers handle values that cannot be serialized deterministically or safely?
  3. Is a source rename/file move intentionally a new testId, or is preserving identity across renames an expected producer responsibility?
  4. For retries executed as separate processes and later collapsed into one CTRF test object, should the raw per-process documents and the merged test share one executionId, or should each raw document have its own execution identity and the merged document get another one?
  5. Is attemptId intentionally limited to previous attempts in retryAttempts? The final attempt is represented by the test object, which has no attemptId; that makes it difficult to correlate the final attempt between a raw per-process report and a merged report.

Why this matters for MTP

Mapping an arbitrary framework UID to testId may overstate its stability, while omitting standardized identities prevents CTRF consumers from correlating parameterized rows and retry artifacts. We would like to document and implement the intended model rather than establish a .NET-specific convention.

Possible interpretations include treating every parameter row as a logical test case, or treating the method as the logical test and the row as an execution. We are not prescribing either; a normative recommendation and one worked parameterized/retry example would unblock the mapping.

Ngôn ngữ chính
Không có dữ liệu ngôn ngữ
Star
99
Fork
4
Merge trung bình
1 giờ 21 phút
Pull request đã merge (30 ngày)
1

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của ctrf-io/ctrf

Tất cả issue của ctrf-io/ctrf

Issue tương tự

Thêm issue về Documentation

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.