Copilot CLI: confirm the agentStop hook sends the agent back on a broken test

Open Beginner friendly
#135 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Active
Tech stack
github, node.js, typescript

Research direction

Read docs/agent-installation.md#github-copilot first, then run npx "@diffci.com/diffci@latest" init in a scratch repo with a test suite and inspect the generated .github/hooks/diffci.json to see how the agentStop handler invokes diffci agent-stop --agent copilot. Run each scenario in Copilot CLI and, if you can, wrap the hook command to capture its stdin payload. Done means a posted report covering Copilot CLI version, OS, step 3 and 4 outcomes, the redacted stdin payload, and any deviation from the docs.

Written by the indexing model from the issue text.

Description

help wanted integration

Part of #132. Nobody has run DiffCI's GitHub Copilot setup in a real session yet. This is the first check, and the one that matters most: does Copilot CLI's agentStop hook send the agent back when its change breaks a test?

You need: Copilot CLI signed in to an account with Copilot, Node.js, and any small repository with a test suite (JS/TS, Python or Go).

Steps

  1. In the repository, run npx "@diffci.com/diffci@latest" init and commit nothing yet. Check that .github/hooks/diffci.json exists with an agentStop handler.
  2. Start Copilot CLI in the repository and ask it to make a change that breaks an existing test (for example, change a function's return value without updating its test).
  3. When the agent ends its turn, the hook runs diffci agent-stop --agent copilot. Expected: the agent gets another turn, with DiffCI's failing tests as the reason, and fixes them. After 3 failed attempts in a session, it should hand back to you.
  4. Repeat with a change that keeps the tests passing. Expected: the agent stops normally, with no extra turn.

Please post

  • Copilot CLI version, OS, and the language of the repository.
  • What happened in steps 3 and 4: did the agent continue, what reason did it see, and did it stop when it should?
  • The stdin payload the hook received, if you can capture it (for example by wrapping the command in the hook file to write stdin to a file). Remove anything private first.
  • Anything that didn't match docs/agent-installation.md#github-copilot.

A report is a contribution: it is credited like a PR, and sustained work on the Copilot integration is how the Area maintainer tier is offered (CONTRIBUTING.md).

Dominant language
TypeScript
Stars
0
Forks
4
Avg merge
18h 41m
Merged PRs (30d)
48

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from DiffCI/core

All issues in DiffCI/core

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.