[BUG] verify gate: bite check's fixed 180s timeout is unreachable for internal/controller, and scores the timeout as a pass
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
Research direction
Start with the bite check in pkg/foreman/agent/tools/gate_job_template.yaml and compare its timeout and result handling with the gate deadline behavior in #1748 and infra anomaly handling in #1259. Reproduce with an internal/controller test package, which the report says takes 536.7 seconds under make test. Done means unfinished tests are reported as GATE-WARN, and captured logs retain the gate verdict.
Written by the indexing model from the issue text.
Description
Bug Description
The verify gate's bite check runs the changed test packages with a fixed
-timeout=180s:
go test -count=1 -timeout=180s $test_pkgs
pkg/foreman/agent/tools/gate_job_template.yaml
internal/controller is an envtest package that cannot finish in 180s. In the
same gate run that produced this report, the gate's own make test measured it
at 536.7s. So any coder branch whose diff touches a test file under
internal/controller is guaranteed to panic on timeout inside the bite check.
Two consequences, and the second is the worse one.
1. The bite check's signal for that package is destroyed. A timeout exits
non-zero, so bite_rc=1, which the script reads as "tests fail against
pre-change production (biting test, good)". The package is scored as biting
without any test having actually run to completion. That is a false pass, and
it is silent.
2. The panic dump evicts the gate's own diagnostics. A go test timeout in
an envtest package prints a goroutine dump of every live goroutine, thousands
of lines. That is captured into
AgenticTask.status.result.extra.modelExtra.logTail, which is size capped, so
the dump truncates everything after it, including the final GATE PASS /
GATE FAIL line and the per-hunk mutation coverage output. On the run that
prompted this, the task ended GATE-FAIL and the log contains no reachable
statement of what failed.
Steps to Reproduce
- Run any Foreman issue-fix workload whose diff adds or changes a test file
underinternal/controller, with the bite check enabled. - Observe
=== bite check (base=main) ===in the gate log. - Observe
panic: test timed out after 3m0sfollowed by a goroutine dump. - Observe that
logTailends inside the dump.
Reproduced locally against the same merge base on branch
foreman/wl-1759-serial/issue-1759 (now PR #1789), producing byte for byte the
same bite check output as the gate.
Expected Behavior
The bite check should be able to distinguish "the reverted tests failed, so they
bite" from "the reverted tests never finished". Concretely:
- The timeout should be at least the time the same packages took under
make testin the same run, or settable the way the gate deadline became
settable in #1748. - A timeout or build hang should be a loud skip (
GATE-WARN), the way other
infra anomalies already are per #1259, rather than counting as a bite. - The captured
logTailshould keep the verdict. Truncating from the front, or
filtering goroutine dumps before capture, would keep the gate's own output
readable when a test times out.
Actual Behavior
A fixed 180s timeout, a timeout scored as a successful bite, and a log whose
verdict is unreachable.
Environment
- LLMKube main as of 5db0f7c2e
- Gate Job from
pkg/foreman/agent/tools/gate_job_template.yaml - Observed on AgenticTask
wl-1759-serial-verify-1759, elapsed 1561s
Additional Context
Related: #1748 made the gate Job deadline settable and mapped DeadlineExceeded
to GATE-ERROR. This is the same class of problem one level down, at the
per-package go test timeout inside the bite check.
Note that the bite check is also not a meaningful probe for a pure move
refactor: when production code is relocated behind delegating shims, the
package's behavior is unchanged by design, so a whole-diff revert leaves its
tests passing. That is a separate discussion from this timeout, but it showed up
on the same run.
Assisted-by: Claude Code reproduced the timeout locally and drafted this report.
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day