[Bug]: Failover combo does not move on when the native Codex upstream declines with server_is_overloaded before any output

Open
  • Is something wrong?
#8,107 2 comments 1 reaction 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
typescript
Domain
backend-api-design

Research direction

Start with poolPreOutputRefusalHold() in src/server/responses/core-codex-account.ts and the failure handling around comboFailureDecision() in src/server/responses/core-combo.ts. Trace how combo attempts and main auth affect whether a pre-output refusal is replayable, then find and run the relevant tests. Done means a structured 5xx before output allows the combo to try its next target, while a refusal after output still stops.

Written by the indexing model from the issue text.

Description

account-pool bug
Description

A failover combo whose first target is the native Codex route does not move to its next target when the upstream declines with server_is_overloaded (503) before any output. The combo returns the 503 to the client. Codex then reports "Selected model is at capacity. Please try a different model." and ends the turn.

This is the combo and main-account counterpart of #6969. The pre-output refusal hold from #6969 makes such a decline replayable, but it is enabled only for non-combo requests on pool / main-pool auth.

Steps to reproduce
  1. Run OpenCodex 2.95.0 with the openai provider in codexAccountMode: direct, authMode: forward (the main ChatGPT account, no pool).
  2. Define a failover combo gpt-6.1-sol with targets [openai/gpt-6.1-sol, deepseek/deepseek-flash] (any second provider shows the same result).
  3. Send streaming Responses requests (Codex CLI and other Responses clients) to gpt-6.1-sol while the ChatGPT backend is at capacity.
Expected

A decline before any output is safe to replay. The combo should try its next target, because comboFailureDecision() already returns hop for 5xx and for server_is_overloaded.

Actual

Each failed request has one attempt and the combo stops:

  • request: provider: "combo", status: 503, firstOutputMs: null
  • attempt: provider: "openai", status: 503, errorCode: "server_is_overloaded", sendCount: 1, recoveryKinds: [], failureStage: "protocol-prelude", failureCause: "upstream-declined"
  • codexWsStage: sent: true, relayedEvents: 4, terminalEvent: "error"
  • routeDecision.candidates[1] (the second target): exclusions: [{ code: "not-selected" }]

On 2026-11-02, 116 of 488 combo requests to the native route failed this way. Before 2026-11-01 the rate was below 1%.

As we read the code:

  1. poolPreOutputRefusalHold() (src/server/responses/core-codex-account.ts:174) returns undefined when comboAttempt is true. It also requires usesCodexForwardPoolAuth(), which is true only for pool and main-pool auth contexts, not main.
  2. Without the hold, the empty lifecycle frames are relayed before the refusal, so the failure is a post-send WebSocket failure and is non-replayable.
  3. In src/server/responses/core-combo.ts (around line 1381), failure.nonReplayable makes the decision stop before comboFailureDecision() is asked.

The same code is on main at 684ad75d2.

Suggested change

Enable the #6969 pre-output refusal hold for combo attempts and for the main auth context. Then a structured 5xx that arrives before any output settles as a replayable status, and the combo moves on. A refusal after any output should still stop, as it does now.

Environment
  • OpenCodex 2.95.0 (Bun 1.4.3), Linux, ocx start --port 10100 behind a TLS reverse proxy
  • Clients: Codex CLI and a Responses client, both streaming
  • openai provider: codexAccountMode: direct, authMode: forward; failover combos
Impact

For agent workloads every transient ChatGPT overload becomes a hard turn failure, although the combo has a healthy next target. Codex CLI review runs failed 61 times on 2026-11-02 for this reason.

Dominant language
TypeScript
Stars
16.9k
Forks
1.3k
Avg merge
18h 28m
Merged PRs (30d)
697

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lidge-jun/opencodex

All issues in lidge-jun/opencodex

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.