[ci-scan-feedback] Prevent shared-Helix-log catch-all KBE signatures

Open Beginner friendly
#133,010 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Active
Tech stack
github-actions

Research direction

Read .github/workflows/shared/create-kbe.instructions.md, especially the Rules list around L445 and the Bad vs good signatures table around L556. Compare the proposed guidance with KBE #131449 and tracker #128742; it is done when the instructions cover shared Helix-log catch-all signatures and explain how to anchor or exclude suite-wide output.

Written by the indexing model from the issue text.

Description

agentic-workflows area-Infrastructure untriaged

Triggering signals

  • KBE #131449 ([ci-scan], open) — MEMBER janvorli (2026-08-07): "This issue seems to be a kitchen sink for all kinds of profiler tests failures." (comment)
  • KBE #131449 — MEMBER noahfalk (2026-08-22): "Previously the KBE matched the Helix console log which runs many tests. Whenever some other profiler test failed the full log would still contain the line saying the nongcheap.cmd test ran as well as the error message, just that the error was part of a different test's output." (comment)
  • Tracker #128742 — MEMBER noahfalk (2026-08-22): "the KBE generation logic either ideally either needs to avoid matching error message in the console log, or we need to capture a single console output expression that both indicates an error and is test-specific." (comment)

Rubric finding: KBE #131449's array-form ErrorMessage paired Running test: profiler/gc/nongcheap/nongcheap.cmd (announcement line, present in the profiler suite's single shared Helix console log for every test) with Profiler tests are expected to contain the text 'PROFILER TEST PASSES' (a shared-harness error any profiler test can print). Both lines coexist in the same console log, so the signature matched whenever any profiler/** test failed — a catch-all.

Proposed edits

  • .github/workflows/shared/create-kbe.instructions.md (array-form "Rules" list, ~L445) — new rule: do not pair a per-run announcement line with a non-test-specific error line when a leg runs many tests through one shared Helix console log; anchor every element to the failing test's own output, or set ExcludeConsoleLog: true. Ties directly to noahfalk's console-log-matching feedback.
  • .github/workflows/shared/create-kbe.instructions.md ("Bad vs good signatures" table, ~L556) — new Bad/Good row using the #131449 signature as the concrete example.

Expected behavior change

The next scanner run will not emit array-form KBEs whose elements individually match a shared multi-test Helix console log. Signatures that can only be anchored to suite-wide output will either be tightened to the failing test's own lines or set ExcludeConsoleLog: true, avoiding the profiler-style catch-all KBE that buckets unrelated failures.

[!NOTE]
This PR body was generated by the ci-failure-scan-feedback agentic workflow (AI/Copilot-generated).

Generated by CI Outer-Loop Failure Scanner — Feedback · opus48 · 259.5 AIC · ⌖ 25.4 AIC · ⊞ 20.6K ·


[!NOTE]
This was originally intended as a pull request, but the git push operation failed.

Original error: The process '/usr/bin/git' failed with exit code 1

Workflow Run: View run details and download bundle artifact

The bundle file is available in the agent artifact in the workflow run linked above.

To create a pull request with the changes:

# Download the artifact from the workflow run
gh run download 33510401456 -n agent -D /tmp/agent-33510401456

# Fetch the bundle into a temporary ref, then update the local branch
git fetch /tmp/agent-33510401456/aw-ci-scan-feedback-kbe-shared-helix-log-catchall.bundle refs/heads/ci-scan-feedback/kbe-shared-helix-log-catchall:refs/bundles/create-pr-ci-scan-feedback-kbe-shared-helix-log-catchall-77d6bd95fa74a85b-75f93697
git update-ref refs/heads/ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b refs/bundles/create-pr-ci-scan-feedback-kbe-shared-helix-log-catchall-77d6bd95fa74a85b-75f93697
git checkout ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d refs/bundles/create-pr-ci-scan-feedback-kbe-shared-helix-log-catchall-77d6bd95fa74a85b-75f93697

# Push the branch to origin
git push origin ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b

# Create the pull request
gh pr create --title '[ci-scan-feedback] Prevent shared-Helix-log catch-all KBE signatures' --base main --head ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b --repo dotnet/runtime
Dominant language
C#
Stars
18.3k
Forks
5.6k
Avg merge
2d 19h
Merged PRs (30d)
589

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dotnet/runtime

All issues in dotnet/runtime

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.