[ci-scan-feedback] Prevent shared-Helix-log catch-all KBE signatures
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 72/100
- Issue type
- Documentation
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- github-actions
- Domain
- ci-cd, documentation
Research direction
Read .github/workflows/shared/create-kbe.instructions.md, especially the Rules list around L445 and the Bad vs good signatures table around L556. Compare the proposed guidance with KBE #131449 and tracker #128742; it is done when the instructions cover shared Helix-log catch-all signatures and explain how to anchor or exclude suite-wide output.
Written by the indexing model from the issue text.
Description
Triggering signals
- KBE #131449 (
[ci-scan], open) — MEMBER janvorli (2026-08-07): "This issue seems to be a kitchen sink for all kinds of profiler tests failures." (comment) - KBE #131449 — MEMBER noahfalk (2026-08-22): "Previously the KBE matched the Helix console log which runs many tests. Whenever some other profiler test failed the full log would still contain the line saying the nongcheap.cmd test ran as well as the error message, just that the error was part of a different test's output." (comment)
- Tracker #128742 — MEMBER noahfalk (2026-08-22): "the KBE generation logic either ideally either needs to avoid matching error message in the console log, or we need to capture a single console output expression that both indicates an error and is test-specific." (comment)
Rubric finding: KBE #131449's array-form ErrorMessage paired Running test: profiler/gc/nongcheap/nongcheap.cmd (announcement line, present in the profiler suite's single shared Helix console log for every test) with Profiler tests are expected to contain the text 'PROFILER TEST PASSES' (a shared-harness error any profiler test can print). Both lines coexist in the same console log, so the signature matched whenever any profiler/** test failed — a catch-all.
Proposed edits
.github/workflows/shared/create-kbe.instructions.md(array-form "Rules" list, ~L445) — new rule: do not pair a per-run announcement line with a non-test-specific error line when a leg runs many tests through one shared Helix console log; anchor every element to the failing test's own output, or setExcludeConsoleLog: true. Ties directly to noahfalk's console-log-matching feedback..github/workflows/shared/create-kbe.instructions.md("Bad vs good signatures" table, ~L556) — new Bad/Good row using the #131449 signature as the concrete example.
Expected behavior change
The next scanner run will not emit array-form KBEs whose elements individually match a shared multi-test Helix console log. Signatures that can only be anchored to suite-wide output will either be tightened to the failing test's own lines or set ExcludeConsoleLog: true, avoiding the profiler-style catch-all KBE that buckets unrelated failures.
[!NOTE]
This PR body was generated by theci-failure-scan-feedbackagentic workflow (AI/Copilot-generated).
Generated by CI Outer-Loop Failure Scanner — Feedback · opus48 · 259.5 AIC · ⌖ 25.4 AIC · ⊞ 20.6K · ◷
[!NOTE]
This was originally intended as a pull request, but the git push operation failed.Original error: The process '/usr/bin/git' failed with exit code 1
Workflow Run: View run details and download bundle artifact
The bundle file is available in the
agentartifact in the workflow run linked above.
To create a pull request with the changes:
# Download the artifact from the workflow run
gh run download 33510401456 -n agent -D /tmp/agent-33510401456
# Fetch the bundle into a temporary ref, then update the local branch
git fetch /tmp/agent-33510401456/aw-ci-scan-feedback-kbe-shared-helix-log-catchall.bundle refs/heads/ci-scan-feedback/kbe-shared-helix-log-catchall:refs/bundles/create-pr-ci-scan-feedback-kbe-shared-helix-log-catchall-77d6bd95fa74a85b-75f93697
git update-ref refs/heads/ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b refs/bundles/create-pr-ci-scan-feedback-kbe-shared-helix-log-catchall-77d6bd95fa74a85b-75f93697
git checkout ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d refs/bundles/create-pr-ci-scan-feedback-kbe-shared-helix-log-catchall-77d6bd95fa74a85b-75f93697
# Push the branch to origin
git push origin ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b
# Create the pull request
gh pr create --title '[ci-scan-feedback] Prevent shared-Helix-log catch-all KBE signatures' --base main --head ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b --repo dotnet/runtime
- Dominant language
- C#
- Stars
- 18.3k
- Forks
- 5.6k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 589
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from dotnet/runtime
-
agentic-workflows untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
area-System.Reflection blocking-clean-ci-optional Known Build Error os-mac-os-x untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
area-CodeGen-coreclr untriaged
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
agentic-workflows untriaged
Difficulty 1/5 Under an hour Newbie friendliness 78/100
-
area-VM-meta-mono untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Similar issues
-
type/automation type/tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
t/bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
ci-failure-cause test-failure
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
area:auth FE mvp P3
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
klasolsson81/jobbliggaren#1788 ·