Code review skill produces wildly different results between Claude and Codex
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 47/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- typescript
- Domain
- developer-experience
Research direction
The report describes inconsistent findings from /ce-code-review in Codex and Claude and mentions a possible connection to issue #799, but names no implementation files or tests. Start by reading the skill implementation and issue #799, then compare how reviewer outputs are collected and synthesized across the two integrations. Done means the same diff produces a consistent, adequately reconciled report, with a regression check covering the reported behavior.
Written by the indexing model from the issue text.
Description
Hi there,
First of all, thank you for creating such a great plugin. I've been using it for some time now and it's been great.
I've primarily been a Claude user, but lately I've been testing Codex and I've noticed a big difference in how the /ce-code-review skill behaves between the two.
When used against the same task/commit diff, Codex tends to return single-digit findings, while Claude usually produces double-digit findings. The difference between the two is pretty significant.
When I asked Codex why there was such a difference between the results, it said:
I did not spawn the full applicable reviewer set for the final diff.
I did not create/read a fresh run artifact and then synthesize from every reviewer output.
I over-applied the “validate and drop weak findings” mindset before showing you the raw-to-final mapping.
Claude Code has a more direct implementation path for this CE skill; in Codex I had to manually emulate it with spawn_agent, and I under-emulated it.
This isn't a one-off, it consistently happens every time I run the skill. Is this a bug or a limitation of Codex? I'm also not sure if it's related to this https://github.com/EveryInc/compound-engineering-plugin/issues/799 ?
Edit: here's some more comments from Codex:
Why results diverged:
Claude artifacts include raw reviewer output, not just final merged findings. Many items are testing_gaps or residual_risks, not primary findings. My final report only surfaced the main actionable findings and summarized coverage too tightly.
A lot of Claude’s raw findings violate the skill’s strict merge schema: severity values like high/medium/low, confidence values like 90/95/80. Under the skill’s Stage 5 validation, those are supposed to be dropped or normalized before final synthesis. I dropped some instead of salvaging/normalizing useful content.
I demoted or omitted several “coverage-only” items that Claude preserved, ....
Reviewer nondeterminism is real here. Different harness, model tiering, and agent implementations produced different issue discovery. But that does not fully excuse it: several Claude findings are valid and I should have caught or at least surfaced them as testing gaps/residual risks.
I also compressed the final report too much after applying the one safe fix. The skill asks for richer Coverage, Deployment Notes, and residual/actionable tables. My final answer was materially thinner than the Claude artifact set.
So the honest answer: I did spawn the required agents, but my synthesis pass was stricter and less complete than it should have been. The biggest actual miss is not agent coverage; it is that I failed to reconcile valid raw signals into the final report.
- Dominant language
- TypeScript
- Stars
- 25.6k
- Forks
- 2.1k
- Avg merge
- 3h 57m
- Merged PRs (30d)
- 170
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from EveryInc/compound-engineering-plugin
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
EveryInc/compound-engineering-plugin#2085 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
EveryInc/compound-engineering-plugin#795 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 80/100
EveryInc/compound-engineering-plugin#2084 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
EveryInc/compound-engineering-plugin#2083 ·
Maintainers usually reply within 1 day
-
Host support: DeepSeek Harness (dsh)Possibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 3/5 1-2 days Newbie friendliness 48/100
EveryInc/compound-engineering-plugin#2072 ·
Maintainers usually reply within 1 day
All issues in EveryInc/compound-engineering-plugin
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
hurttlocker/o8#6535 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
Maintainers usually reply within 1 day
-
area: backend bug priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 92/100
snapotter-hq/SnapOtter#4856 ·
Maintainers usually reply within 1 day
-
bug follow-up
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
spellshift/realm#2509 ·
Maintainers usually reply within 1 day