skillopt-sleep Codex harvest ingests its own headless replay sessions
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 72/100
Research direction
Start at harvest_codex() and digest_codex_archived_session(), then compare their handling with the Claude, Copilot CLI, and OpenCode transcript harvesters. Trace sessions produced by CodexCliBackend.attempt_with_tools() and add the equivalent exclusion for the shown engine-generated prompt shape. Done means the reproduced archived session is not returned by digest_codex_archived_session() or included by harvest_codex().
Written by the indexing model from the issue text.
Description
Summary
harvest_codex() accepts Codex sessions created by SkillOpt-Sleep itself during headless replay, judging, reflection, and tool-aware attempts. The other transcript harvesters explicitly reject engine-generated sessions, but the Codex source has no equivalent filter.
Reproduction
On main at 79124b37e9a6371e13b753f8bcd7adb1e493ade1, create a Codex archived session using the prompt shape emitted by CodexCliBackend.attempt_with_tools():
Complete the task. Apply the skill and memory rules EXACTLY, including any rule about searching before answering.
# Skill
...
# Memory
...
# Task
...
with a normal user_message and agent_message record. digest_codex_archived_session() returns a SessionDigest, and harvest_codex() includes it.
Expected behavior
Engine-generated Codex sessions should be excluded from harvested user evidence, just as Claude, Copilot CLI, and OpenCode replay/agent sessions are excluded.
Actual behavior
The generated prompt is treated as a real user task. On a later night it can be mined, replayed, and used to update the skill, creating self-training feedback and contaminating the task set.
Impact
Codex-backed runs can learn from their own evaluator/attempt instructions instead of user work, weakening task provenance and potentially reinforcing internal prompt text.
Environment
SkillOpt-Sleep main at 79124b37e9a6371e13b753f8bcd7adb1e493ade1; Python 3.10+; Codex CLI/archived sessions.
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 1.6k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 9
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/SkillOpt
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 5/5 Over a week Newbie friendliness 42/100
-
Difficulty 5/5 Over a week Newbie friendliness 28/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
All issues in microsoft/SkillOpt
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
syfoud/Simulated_Scepter#172 ·
-
A cancelled tests run makes the coverage comment workflow fail and reports it as a red check on main Openarea: ci bug perceived difficulty: 3
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Nitjsefnie-Harness-Commons/daedalus#921 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
EleutherAI/lm-evaluation-harness#4207 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ClickHouse/clickhouse-connect#1057 ·