Bug: tools declared by a custom_agents agent are announced but not callable by the model

Bắt đầu với repro.py và chạy các trường hợp A/B của create_session bằng custom_agents so với available_tools. Theo dõi cách các tùy chọn đó đi qua create_session, sau đó so sánh các công cụ của agent được chọn với các công cụ có thể gọi trong session-state//events.jsonl. Được xem là hoàn thành khi các công cụ đã khai báo có thể được gọi trong lượt của custom agent, hoặc create_session từ chối cấu hình không được hỗ trợ.

Bug: tools declared by a custom_agents agent are announced but not callable by the model

Summary

When a session is created with custom_agents=[...] plus agent="<name>", the tools that agent
declares in its own tools: list are not usable by the model. The runtime reports them as
selected (a subagent.selected event lists them verbatim), but the model behaves as though it has
no tools at all: it makes zero tool calls and states plainly that it has no tool capable of
doing the work.

Passing the identical tool list via available_tools=[...] on the same model, same prompt,
same sandbox works correctly.

The failure is silent. There is no error, no warning, and no rejected tool call — the model simply
answers in prose. A caller that only inspects the final message (or a subagent.selected event)
sees a perfectly healthy-looking session.

Environment

Python SDK	github-copilot-sdk 1.0.11
CLI / server	copilot 1.0.79 (also reproduced on 1.0.78)
Connection	RuntimeConnection.for_uri(...) to a copilot --headless --server process in a container
Models	reproduced with gpt-5.3-codex and gpt-5.4
agent_mode	"autopilot"
Permissions	PermissionHandler.approve_all
OS	server on Linux (Ubuntu 24.04 container); client on Windows

Reproduction

repro.py below runs the same request twice against the same server: once with custom_agents,
once with available_tools. Start a headless server first, then run it.

# a copilot --headless --server on :3000, published to 18097, with a /workspace/repo cwd
docker run -d --rm --name ghcprepro -p 18097:3000 \
  -e COPILOT_SDK_AUTH_TOKEN="$GITHUB_TOKEN" \
  -e COPILOT_CONNECTION_TOKEN="repro123" \
  <image running: copilot --headless --no-auto-update --auth-token-env COPILOT_SDK_AUTH_TOKEN \
                          --no-auto-login --host 0.0.0.0 --port 3000>

python repro.py gpt-5.3-codex
docker exec ghcprepro sh -c 'ls /workspace/repo/'   # which file actually got created?

"""Minimal A/B: custom_agents' declared tools vs the identical list via available_tools."""
import asyncio, sys
from copilot import CopilotClient, RuntimeConnection
from copilot.session import PermissionHandler
from copilot.session_events import SessionEventType

URI, TOKEN = "localhost:18097", "repro123"
MODEL = sys.argv[1] if len(sys.argv) > 1 else "gpt-5.3-codex"
TOOLS = ["builtin:view", "builtin:grep", "builtin:glob",
         "builtin:edit", "builtin:create", "builtin:apply_patch", "builtin:skill"]

WRITER_AGENT = {
    "name": "writer",
    "description": "Creates files on disk.",
    "tools": TOOLS,                       # <-- identical list to available_tools in case B
    "model": MODEL,
    "prompt": "You are the Writer Agent. You create files on disk with your file tools.",
}

ASK = ("Create a file named {name} in the current directory containing exactly the text 'hello'. "
       "Then tell me in one sentence which tool you used, or state plainly that you have no tool "
       "capable of creating a file.")

async def run_case(name: str, *, use_custom_agent: bool) -> str:
    connection = RuntimeConnection.for_uri(URI, connection_token=TOKEN)
    async with CopilotClient(connection=connection, log_level="error") as client:
        kwargs = {"on_permission_request": PermissionHandler.approve_all, "model": MODEL,
                  "streaming": True, "working_directory": "/workspace/repo"}
        if use_custom_agent:
            kwargs["custom_agents"] = [WRITER_AGENT]
            kwargs["agent"] = "writer"
        else:
            kwargs["available_tools"] = TOOLS
        session = await client.create_session(**kwargs)

        done, text = asyncio.Event(), []
        def on_event(e):
            if e.type == SessionEventType.ASSISTANT_MESSAGE: text.append(e.data.content or "")
            elif e.type in (SessionEventType.SESSION_IDLE, SessionEventType.SESSION_ERROR): done.set()
        session.on(on_event)
        await session.send(ASK.format(name=name), agent_mode="autopilot")
        await asyncio.wait_for(done.wait(), timeout=120)
        return f"session={session.session_id} :: {' '.join(text).strip()[:200]}"

async def main():
    print("A custom_agents  ->", await run_case("case_a.txt", use_custom_agent=True))
    print("B available_tools->", await run_case("case_b.txt", use_custom_agent=False))

asyncio.run(main())

Actual result

A custom_agents  -> session=41113715-... :: I have no tool capable of creating a file.
B available_tools-> session=396f85a7-... :: Creating `case_b.txt` now ... I used the `apply_patch`
                                            tool to create `case_b.txt` with exactly `hello`.

$ ls /workspace/repo/
case_b.txt        # case_a.txt was never created

Case A reproduces on every attempt (3/3 trials, plus both models above).

The contradiction, from the server's own session log

~/.copilot/session-state/<session-id>/events.jsonl for the failing case A session announces
exactly the tools that the model then cannot use:

{"type":"subagent.selected","data":{"agentName":"writer","agentDisplayName":"writer",
 "tools":["builtin:view","builtin:grep","builtin:glob","builtin:edit","builtin:create",
          "builtin:apply_patch","builtin:skill"]}}

Tool invocations recorded in each session:

session	tool.execution_start entries
A (custom_agents)	none at all
B (available_tools)	apply_patch ×1

So the agent's tool set is resolved and reported, but never reaches the model's callable tool set.

Expected result

An agent's declared tools: should be callable by the model in that agent's turn — equivalent to
passing the same list via available_tools. Case A should create case_a.txt.

Failing that, a session whose agent declares tools the runtime cannot expose should raise an error
at create_session, rather than silently producing a tool-less agent.

Impact

This is expensive to diagnose because every observable signal says the session is healthy: the
agent is selected, the declared tools are echoed back, agent_mode is autopilot, permissions are
auto-approved, and the model returns a confident, well-formed answer. Only the filesystem (or the
absence of tool.execution_start events) reveals that nothing happened.

In our pipeline this manifested as agents that "completed" substantial work in ~18 seconds while
writing nothing to disk, and as automated fix-up steps that never repaired anything. Because each
affected stage then failed a downstream check for an unrelated-looking reason, it took a long time
to trace back to tool availability.

Workaround

Do not use custom_agents for any agent that needs tools. Pass the tool list via
available_tools on create_session, and supply the agent's instructions as a normal system
message instead of an agent definition.

Secondary observation (possibly intended, but surprising)

builtin:create alone is not sufficient for file creation — with
[view, grep, glob, edit, create] the model reports it has no way to create a file, and creation
only succeeds once builtin:apply_patch is included (the model then uses apply_patch). If
create is not independently usable, it would help for its absence/aliasing to be documented, or
for create to be rejected as unknown rather than accepted silently.