Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Codex rollout reader drops raw reasoning from open-model servers

Open
  • Is something wrong?
#1,017 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
data

Research direction

Read engine/resources/agent_rollout/codex.py, especially extract_codex_reasoning_summaries, and the rollout tests under tests/engine/resources/agent_rollout/. Use the provided JSONL reproduction to check how reasoning items with content and summary are parsed. Done means the assistant message has reasoning_content from content, with summary as fallback, and the existing rollout tests still pass.

Written by the indexing model from the issue text.

Description

bug triaged
Priority Level

Medium (Annoying but has workaround)

Describe the bug

AgentRolloutSeedSource with format=codex drops the reasoning of Codex sessions that ran against an open model served by vLLM.
extract_codex_reasoning_summaries in engine/resources/agent_rollout/codex.py reads only the summary field of a reasoning response item.
vLLM's Responses API returns reasoning as content: [{"type": "reasoning_text", ...}] with an empty summary, and Codex writes that item to the rollout file unchanged.
The resulting assistant messages have no reasoning_content, so the reasoning is missing from the seed data.

Steps/Code to reproduce bug

Reproduced on main (c05a87e) with Python 3.14.3 and dependencies installed with uv sync --all-packages --frozen.
No model calls are needed.
The code calls CodexAgentRolloutFormatHandler, the handler that AgentRolloutSeedSource(format="codex") uses (registered in agent_rollout/registry.py).

import json
import tempfile
from pathlib import Path

from data_designer.engine.resources.agent_rollout.codex import CodexAgentRolloutFormatHandler

root = Path(tempfile.mkdtemp())
rows = [
    {"type": "session_meta", "payload": {"id": "sess-1"}},
    {"type": "response_item", "payload": {"type": "message", "role": "user", "content": [{"type": "input_text", "text": "What is 2+2?"}]}},
    {"type": "response_item", "payload": {"type": "reasoning", "summary": [], "content": [{"type": "reasoning_text", "text": "2+2 is 4."}], "encrypted_content": None}},
    {"type": "response_item", "payload": {"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "4"}]}},
]
(root / "rollout-repro.jsonl").write_text("\n".join(json.dumps(r) for r in rows) + "\n")
record = CodexAgentRolloutFormatHandler().parse_file(root_path=root, relative_path="rollout-repro.jsonl")[0]
print(next(m for m in record.messages if m["role"] == "assistant").get("reasoning_content"))

Output: None

Expected behavior

The assistant message has reasoning_content == "2+2 is 4.".
When a reasoning item has text in both content and summary, the reader should prefer content (the full reasoning) and fall back to summary, OpenAI-hosted models do not return raw reasoning content, so their rollouts keep the current behavior.

Agent Diagnostic / Prior Investigation
  • Searched open and closed issues, PRs, and discussions for "codex reasoning", "codex rollout", "reasoning_content", "reasoning_text", "agent_rollout", and "AgentRolloutSeedSource": no existing report.
    #374 also concerns vLLM reasoning fields, but in model client response parsing, not in rollout ingestion.
  • Searched the docs: the agent rollout ingestion page (agent-rollout-ingestion.mdx) does not mention reasoning, so it does not say which Codex reasoning fields are read.
  • End to end: Codex CLI 0.159.2 against vLLM 0.30.0 serving nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, two turns with tool calls, ingested with Data Designer 0.9.2.
    The rollout file had 4 reasoning items with reasoning_text content and an empty summary.
    The reader set reasoning_content on 0 of 4 corresponding assistant messages.
  • Workaround: rewrite the rollout files to copy content into summary before ingestion.
Additional context

I have a fix ready: a small change in codex.py (+11/−6 lines) that reads content first and falls back to summary, plus one unit test.
On main, tests/engine/resources/agent_rollout/ passes (49 tests) with the fix applied, and the new test fails without it.
I will open the PR once this issue is triaged.

Checklist
  • I reproduced this issue or provided a minimal example
  • I searched the docs/issues myself, or had my agent do so
  • If I used an agent, I included its diagnostics above
Dominant language
Python
Stars
2.3k
Forks
227
Avg merge
1d 12h
Merged PRs (30d)
52

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA-NeMo/DataDesigner

All issues in NVIDIA-NeMo/DataDesigner

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.