app-server: silent turn/completed(last_agent_message=null) when run_turn early-returns after compaction failure

Open Beginner friendly
#25,619 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
74/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
rust
Domain
api, backend

Research direction

Start in codex-rs/core/src/session/turn.rs::run_turn and compare the pre-sampling and mid-turn compaction failure branches with the existing catch-all around L394-L411. Verify the related convention in codex-rs/core/src/compact.rs:242-244. Done means both early-return paths emit a wire-visible EventMsg::Error before task_complete, allowing app-server clients to distinguish failure from an empty response.

Written by the indexing model from the issue text.

Description

app-server bug CLI context

What issue are you seeing?

Several early-return paths in codex-rs/core/src/session/turn.rs::run_turn make the function return None (which yields a task_complete with last_agent_message=null, zero last_token_usage, and no time_to_first_token_ms), without emitting an EventMsg::Error to the wire/rollout. The companion emit_turn_error_lifecycle call in those paths only notifies in-process extension contributors; tracing::error!() only writes to the local log. JSON-RPC clients connected to a codex app-server --listen see a normal-looking completed turn and cannot tell that the model was never sampled.

From a programmatic client's view, the WebSocket exchange is:

-> turn/start { threadId, input: [...] }
<- turn/started
<- token/count   (total_token_usage.total_tokens == model_context_window if set_token_usage_full has fired previously)
<- token/count
<- turn/completed { turn: { id, status: "completed" },
                    last_agent_message: null,
                    time_to_first_token_ms: null,
                    duration_ms: 2_000-12_000 }

No error event. No item events. Indistinguishable on the wire from a normal completion in which the model voluntarily returned an empty assistant message.

Affected paths (HEAD checkout)

In codex-rs/core/src/session/turn.rs::run_turn:

  • The pre-sampling compact branch around L147-L162 (catching run_pre_sampling_compact Err) calls emit_turn_error_lifecycle + tracing::error!("Failed to run pre-sampling compact") then return None;. It does not call sess.send_event(turn_context, EventMsg::Error(...)).
  • The mid-turn auto-compact branch (auto-compact after a sampling result, ~L290-L316) follows the same pattern: extension lifecycle + tracing log + return None;, no wire-visible event.

By contrast, the existing catch-all at the bottom of the main loop (~L394-L411) does emit the wire-visible event:

Err(e) => {
    info!("Turn error: {e:#}");
    let error = e.to_codex_protocol_error();
    sess.emit_turn_error_lifecycle(turn_context.as_ref(), error.clone()).await;
    if error == CodexErrorInfo::UsageLimitExceeded { /* ... */ }
    let event = EventMsg::Error(e.to_error_event(/*message_prefix*/ None));
    sess.send_event(&turn_context, event).await;
    break;
}

And codex-rs/core/src/compact.rs:242-244 follows the same convention before returning Err(ContextWindowExceeded). So the established pattern across the codebase is "emit EventMsg::Error before failing the turn" — the two paths above just slipped through.

What steps can reproduce the bug?

  1. Bring up a codex app-server --listen unix:///tmp/codex.sock whose persistent session is in the "context-full" state, i.e. one of its prior turns triggered Session::set_total_tokens_full (either via the ContextWindowExceeded branch in try_run_sampling_request at core/src/session/turn.rs:985-988, or via auto-compact exhaustion in core/src/compact.rs:242). After that fires once, state.history.token_info is pinned at total_tokens == model_context_window until a successful compaction resets it.
  2. From any JSON-RPC client (e.g. a small Python WebSocket script or websocat), drive:
    • initialize
    • thread/resume { "threadId": "<id>", "excludeTurns": true, "persistExtendedHistory": true }
    • turn/start { "threadId": "<id>", "input": [{ "type": "text", "text": "<<no reply needed>>", "text_elements": [] }], "approvalPolicy": "on-request" }
  3. Read frames until turn/completed for the new turn_id.

Observed: turn/completed with turn.status = "completed", last_agent_message = null, all-zero last_token_usage, no error event in between. The rollout JSONL records task_started, turn_context, the response_item for the user message, the user_message event, two token_count events (with the synthetic total_token_usage.total_tokens == model_context_window), then task_complete. Nothing else.

Observation that surfaced this

On a long-lived persistent session (open for several weeks, ~85k rollout lines, ~138MB):

  • Several consecutive turn/start requests to the app-server over multiple hours produced silent task_complete(null) turns (durations 2.1 s - 11.4 s, no ttft, no Error).
  • Recovery happened only when a user typed input into a separate codex resume --last process attached to the same thread. That second process has its own in-memory Session and apparently completed a compaction successfully; subsequent turns through it sampled normally.
  • The app-server's in-memory session state remained context-full for the entire failing window. Rate limits were fine (primary 2%, secondary 13%, rate_limit_reached_type: null).

What is the expected behavior?

When run_turn is going to return None because of a pre-sampling or mid-turn compaction failure (or any path where the model was not actually invoked), the app-server should emit a wire-visible EventMsg::Error (or a new variant such as EventMsg::TurnFailed) before the task_complete, so a JSON-RPC client can detect that the turn was not sampled. The task_complete itself could additionally carry a richer status than "completed" in those cases (e.g. "aborted"/"failed"/"context_full"), but even the simpler fix — propagating the existing EventMsg::Error — would close the observability gap.

Minimal illustrative patch

Aligns the two missing paths to the convention already used at turn.rs:394-411 and compact.rs:242-244:

// core/src/session/turn.rs - pre-sampling compact branch
if let Err(err) = run_pre_sampling_compact(&sess, &turn_context, &mut client_session).await {
    let error = err.to_codex_protocol_error();
    sess.emit_turn_error_lifecycle(turn_context.as_ref(), error.clone()).await;
    if error == CodexErrorInfo::UsageLimitExceeded
        && let Err(e2) = sess.goal_runtime_apply(GoalRuntimeEvent::UsageLimitReached {
            turn_context: turn_context.as_ref(),
        }).await
    {
        warn!("failed to usage-limit active goal after usage-limit error: {e2}");
    }
+   let event = EventMsg::Error(err.to_error_event(/*message_prefix*/ None));
+   sess.send_event(&turn_context, event).await;
    error!("Failed to run pre-sampling compact");
    return None;
}

And the analogous insertion in the mid-turn run_auto_compact failure branch a few hundred lines lower. Both should adopt the e.to_codex_protocol_error() + e.to_error_event(None) pair already in use at L394-L411 (ownership of err allows both calls, same as the existing pattern).

Additional information

  • Why this is a bug and not a feature: extension contributors and tracing logs are not a substitute for the JSON-RPC surface. Any client that consumes codex app-server programmatically — IDE integrations, codex exec, our JSON-RPC dispatcher — has no robust way today to distinguish "model produced an empty response" from "model was never invoked". The current behavior also disagrees with itself: the catch-all path at L394 emits EventMsg::Error; the two paths above don't. Two-thirds of the same handler family emits an observable failure signal; one-third silently swallows it.
  • Related issues (similar symptom, different surface or root cause):
    • #24536 - codex exec can silently complete empty when configured MCP tools are deferred behind tool_search (same observable from a different cause; a generic wire-level failure signal would help diagnostics there too).
    • #16872 - app-server websocket shared-thread turns complete but rollout never materializes (adjacent symptom in the same area).
    • #7808 - Running out of room in the Codex context window is immediately fatal to that chat thread (the human-facing UX of the same underlying state).
  • Environment:
    • Codex binary: codex 0.125.0 was the persistent app-server at the time of the observation (running as a systemd unit, codex app-server --listen unix:///run/.../app.sock); the interactive sibling was codex 0.134.0 via codex resume --last. Source examined: HEAD of https://github.com/openai/codex.git.
    • OS: Linux (Debian, Linux 6.12.85+deb13-amd64).
    • Protocol used by the client: v2 JSON-RPC over Unix-socket WebSocket (the client does a manual HTTP Upgrade handshake).
  • Investigation context: this was found while hardening our JSON-RPC dispatcher against the symptom. The full investigation notes (with the per-turn event sequences and code-level tracing) are local; happy to attach a sanitized excerpt if useful. The client-side fix is independent (it will start to declare processed=false based on last_agent_message, last_token_usage, item-event count, and the very EventMsg::Error this issue proposes to emit), but a wire-level signal from Codex would let every programmatic client benefit, not just ours.
  • Not asking for: a redesign of the context-full state machine. Even keeping the current pinned token_info after set_total_tokens_full, just emitting EventMsg::Error (or equivalent) on the early-return paths would already make this diagnosable and correctable from the client side.
Dominant language
Rust
Stars
125k
Forks
19.5k
Avg merge
1m
Merged PRs (30d)
1k

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from openai/codex

All issues in openai/codex

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.