Bug: tools declared by a custom_agents agent are announced but not callable by the model

Inizia da repro.py ed esegui i suoi casi A/B di create_session usando custom_agents rispetto a available_tools. Traccia il flusso di queste opzioni attraverso create_session, quindi confronta gli strumenti dell’agente selezionato con gli strumenti richiamabili in session-state//events.jsonl. Il lavoro è concluso quando gli strumenti dichiarati sono richiamabili nel turno dell’agente personalizzato, oppure quando create_session rifiuta la configurazione non supportata.

Bug: tools declared by a custom_agents agent are announced but not callable by the model

Summary

When a session is created with custom_agents=[...] plus agent="<name>", the tools that agent
declares in its own tools: list are not usable by the model. The runtime reports them as
selected (a subagent.selected event lists them verbatim), but the model behaves as though it has
no tools at all: it makes zero tool calls and states plainly that it has no tool capable of
doing the work.

Passing the identical tool list via available_tools=[...] on the same model, same prompt,
same sandbox works correctly.

The failure is silent. There is no error, no warning, and no rejected tool call — the model simply
answers in prose. A caller that only inspects the final message (or a subagent.selected event)
sees a perfectly healthy-looking session.

Environment

Reproduction

repro.py below runs the same request twice against the same server: once with custom_agents,
once with available_tools. Start a headless server first, then run it.

# a copilot --headless --server on :3000, published to 18097, with a /workspace/repo cwd
docker run -d --rm --name ghcprepro -p 18097:3000 \
  -e COPILOT_SDK_AUTH_TOKEN="$GITHUB_TOKEN" \
  -e COPILOT_CONNECTION_TOKEN="repro123" \
  <image running: copilot --headless --no-auto-update --auth-token-env COPILOT_SDK_AUTH_TOKEN \
                          --no-auto-login --host 0.0.0.0 --port 3000>

python repro.py gpt-5.3-codex
docker exec ghcprepro sh -c 'ls /workspace/repo/'   # which file actually got created?

"""Minimal A/B: custom_agents' declared tools vs the identical list via available_tools."""
import asyncio, sys
from copilot import CopilotClient, RuntimeConnection
from copilot.session import PermissionHandler
from copilot.session_events import SessionEventType

URI, TOKEN = "localhost:18097", "repro123"
MODEL = sys.argv[1] if len(sys.argv) > 1 else "gpt-5.3-codex"
TOOLS = ["builtin:view", "builtin:grep", "builtin:glob",
         "builtin:edit", "builtin:create", "builtin:apply_patch", "builtin:skill"]

WRITER_AGENT = {
    "name": "writer",
    "description": "Creates files on disk.",
    "tools": TOOLS,                       # <-- identical list to available_tools in case B
    "model": MODEL,
    "prompt": "You are the Writer Agent. You create files on disk with your file tools.",
}

ASK = ("Create a file named {name} in the current directory containing exactly the text 'hello'. "
       "Then tell me in one sentence which tool you used, or state plainly that you have no tool "
       "capable of creating a file.")

async def run_case(name: str, *, use_custom_agent: bool) -> str:
    connection = RuntimeConnection.for_uri(URI, connection_token=TOKEN)
    async with CopilotClient(connection=connection, log_level="error") as client:
        kwargs = {"on_permission_request": PermissionHandler.approve_all, "model": MODEL,
                  "streaming": True, "working_directory": "/workspace/repo"}
        if use_custom_agent:
            kwargs["custom_agents"] = [WRITER_AGENT]
            kwargs["agent"] = "writer"
        else:
            kwargs["available_tools"] = TOOLS
        session = await client.create_session(**kwargs)

        done, text = asyncio.Event(), []
        def on_event(e):
            if e.type == SessionEventType.ASSISTANT_MESSAGE: text.append(e.data.content or "")
            elif e.type in (SessionEventType.SESSION_IDLE, SessionEventType.SESSION_ERROR): done.set()
        session.on(on_event)
        await session.send(ASK.format(name=name), agent_mode="autopilot")
        await asyncio.wait_for(done.wait(), timeout=120)
        return f"session={session.session_id} :: {' '.join(text).strip()[:200]}"

async def main():
    print("A custom_agents  ->", await run_case("case_a.txt", use_custom_agent=True))
    print("B available_tools->", await run_case("case_b.txt", use_custom_agent=False))

asyncio.run(main())

Actual result

A custom_agents  -> session=41113715-... :: I have no tool capable of creating a file.
B available_tools-> session=396f85a7-... :: Creating `case_b.txt` now ... I used the `apply_patch`
                                            tool to create `case_b.txt` with exactly `hello`.

$ ls /workspace/repo/
case_b.txt        # case_a.txt was never created

Case A reproduces on every attempt (3/3 trials, plus both models above).

The contradiction, from the server's own session log

~/.copilot/session-state/<session-id>/events.jsonl for the failing case A session announces
exactly the tools that the model then cannot use:

{"type":"subagent.selected","data":{"agentName":"writer","agentDisplayName":"writer",
 "tools":["builtin:view","builtin:grep","builtin:glob","builtin:edit","builtin:create",
          "builtin:apply_patch","builtin:skill"]}}

Tool invocations recorded in each session:

So the agent's tool set is resolved and reported, but never reaches the model's callable tool set.

Expected result

An agent's declared tools: should be callable by the model in that agent's turn — equivalent to
passing the same list via available_tools. Case A should create case_a.txt.

Failing that, a session whose agent declares tools the runtime cannot expose should raise an error
at create_session, rather than silently producing a tool-less agent.

Impact

This is expensive to diagnose because every observable signal says the session is healthy: the
agent is selected, the declared tools are echoed back, agent_mode is autopilot, permissions are
auto-approved, and the model returns a confident, well-formed answer. Only the filesystem (or the
absence of tool.execution_start events) reveals that nothing happened.

In our pipeline this manifested as agents that "completed" substantial work in ~18 seconds while
writing nothing to disk, and as automated fix-up steps that never repaired anything. Because each
affected stage then failed a downstream check for an unrelated-looking reason, it took a long time
to trace back to tool availability.

Workaround

Do not use custom_agents for any agent that needs tools. Pass the tool list via
available_tools on create_session, and supply the agent's instructions as a normal system
message instead of an agent definition.

Secondary observation (possibly intended, but surprising)

builtin:create alone is not sufficient for file creation — with
[view, grep, glob, edit, create] the model reports it has no way to create a file, and creation
only succeeds once builtin:apply_patch is included (the model then uses apply_patch). If
create is not independently usable, it would help for its absence/aliasing to be documented, or
for create to be rejected as unknown rather than accepted silently.