Copilot CLI: confirm the agentStop hook sends the agent back on a broken test
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 72/100
- Issue type
- Documentation
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- github, node.js, typescript
- Domain
- developer-experience, testing-qa
Research direction
Read docs/agent-installation.md#github-copilot first, then run npx "@diffci.com/diffci@latest" init in a scratch repo with a test suite and inspect the generated .github/hooks/diffci.json to see how the agentStop handler invokes diffci agent-stop --agent copilot. Run each scenario in Copilot CLI and, if you can, wrap the hook command to capture its stdin payload. Done means a posted report covering Copilot CLI version, OS, step 3 and 4 outcomes, the redacted stdin payload, and any deviation from the docs.
Written by the indexing model from the issue text.
Description
Part of #132. Nobody has run DiffCI's GitHub Copilot setup in a real session yet. This is the first check, and the one that matters most: does Copilot CLI's agentStop hook send the agent back when its change breaks a test?
You need: Copilot CLI signed in to an account with Copilot, Node.js, and any small repository with a test suite (JS/TS, Python or Go).
Steps
- In the repository, run
npx "@diffci.com/diffci@latest" initand commit nothing yet. Check that.github/hooks/diffci.jsonexists with anagentStophandler. - Start Copilot CLI in the repository and ask it to make a change that breaks an existing test (for example, change a function's return value without updating its test).
- When the agent ends its turn, the hook runs
diffci agent-stop --agent copilot. Expected: the agent gets another turn, with DiffCI's failing tests as the reason, and fixes them. After 3 failed attempts in a session, it should hand back to you. - Repeat with a change that keeps the tests passing. Expected: the agent stops normally, with no extra turn.
Please post
- Copilot CLI version, OS, and the language of the repository.
- What happened in steps 3 and 4: did the agent continue, what reason did it see, and did it stop when it should?
- The stdin payload the hook received, if you can capture it (for example by wrapping the command in the hook file to write stdin to a file). Remove anything private first.
- Anything that didn't match
docs/agent-installation.md#github-copilot.
A report is a contribution: it is credited like a PR, and sustained work on the Copilot integration is how the Area maintainer tier is offered (CONTRIBUTING.md).
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 4
- Avg merge
- 18h 41m
- Merged PRs (30d)
- 48
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from DiffCI/core
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Planner: splitParts regex is quadratic on a quote followed by many backticks, so the fuzz test fails at randomPossibly taken @FredApps claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
Agent tools: changed_files lists only the new path of a renamed filePossibly taken @FredApps claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
adapter good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
Similar issues
-
agent-ready area: ui-server priority: p3 type: test
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
schlessera/brain-kit#4249 ·
Maintainers usually reply within 1 day
-
documentation tool triaged
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
can1357/oh-my-pi#19608 · 2 comments ·
Maintainers usually reply within 1 day
-
dependencies tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
bcgov/nr-waste-plus#1549 ·
Maintainers usually reply within 1 day
-
enhancement good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
blixvip/solana-token-meta#17 ·
-
enhancement good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
blixvip/solana-tx-parse#18 ·