Bug: A timed-out or refused upstream request is not flagged as an error on any abctl screen
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 74/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go
- Domain
- api, backend, observability
Research direction
Start with authbridge/authlib/listener/forwardproxy/server.go:491-495 and authlib/listener/reverseproxy/server.go:605-611, then trace the response-event handling in authbridge/authlib/usage/usage.go. Reproduce with the 300ms client timeout against the hanging upstream. Done means transport failures produce response-phase events carrying the 502 and an error in both listeners, so the event, errors, status, and latency views reflect the failure.
Written by the indexing model from the issue text.
Description
Summary
When an outbound request fails at the transport level — a timed-out LLM call, connection refused, DNS failure, TLS failure — the client correctly receives a 502, but no response-phase session event is recorded. The failure is therefore invisible on every abctl screen: it cannot be distinguished from a request that is still in flight.
Reproduction
A hanging upstream behind a proxy whose client times out (probe against authlib/listener/forwardproxy, an empty pipeline, Client: &http.Client{Timeout: 300ms}, upstream sleeping 3s):
client saw status: 502
events recorded: 1
phase=request status=0 host="127.0.0.1:59946" error=<nil>
One event, phase=request. No response event, so nothing carries the 502 and nothing carries an error.
Cause
authbridge/authlib/usage/usage.go — the request event is appended at the end of the request phase, but the response event (the one carrying StatusCode and Error) is appended much later. The upstream failure path returns between the two:
authbridge/authlib/listener/forwardproxy/server.go:491-495
resp, err := client.Do(r)
if err != nil {
http.Error(w, `{"error":"bad gateway"}`, http.StatusBadGateway)
return
}
The 502 exists only on the wire to the client, never in the session store. reverseproxy's errorHandler (authlib/listener/reverseproxy/server.go:605-611) has the same shape.
What each screen shows
| Screen | Behaviour |
|---|---|
| Events table | One req row with STATUS blank (statusCell returns "" for StatusCode == 0), DURATION blank, no paired resp row — identical to a request still in flight |
| Usage → ERRORS | Not counted. usage.go:563 sets Errors = 1 only for StatusCode >= 400 or SessionDenied; a timeout is neither |
| Usage → breakdown by status | Absent. The label is added only when StatusCode > 0, so the request falls into the (unlabelled) remainder |
| Usage → latency | Excluded — a zero duration is treated as "not measured" rather than as a failure |
| Detail pane | The request JSON, with no error field and no response |
Why it matters
The request event is counted in Requests, so a burst of timeouts renders as ordinary traffic with a healthy error rate. The usage pane is least informative exactly when an operator is trying to find out whether their upstream is failing — and "STATUS blank" reads as "still waiting", which is the wrong conclusion.
SessionEvent already has an Error field (pipeline.DeriveError), but it is only ever populated on the response path.
Suggested fix
Record a response-phase event with StatusCode: 502 and a populated Error before the early return, in both listeners. That would surface the failure in the events table, the ERRORS metric, and the status breakdown at once, without any change to those consumers.
An Error value distinguishing why the upstream call failed (timeout vs refused vs TLS) would be more useful than the status code alone, since all of them surface as 502 today.
Scope of this report
Verified for proxy-sidecar's forward proxy — the laptop / Claude Code path. reverseproxy looks the same by inspection but was not tested, and envoy-sidecar's ext_proc path was not examined at all, so I cannot say whether it records a response event on upstream failure.
Operating system and architecture
Darwin arm64
Cortex version
dev (traced at b838de5f; both cited files are unmodified from main)
- Dominant language
- Go
- Stars
- 13
- Forks
- 40
- Avg merge
- 12h 9m
- Merged PRs (30d)
- 164
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from rossoctl/cortex
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
nice to have
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
stale
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 84/100
-
enhancement needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
kind/cleanup
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
kubernetes-sigs/kueue#15947 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
sympozium-ai/sympozium#627 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100