Remote AG-UI Bots can't use the browser in chat: `computer_*` calls refused as "unavailable for this headless turn"
維護者通常 1 天內回覆
評估
- 難度
- 3/5
- 預估耗時
- 1-2 天
- 新手友好度
- 68/100
- Issue 類型
- 缺陷
- 描述清晰度
- 基本清楚
- 活躍度
- 活躍
- 技術堆疊
- react, typescript
研究方向
Start with toolkit/src/headless-tools.ts, server/src/copilot.ts, and server/src/index.ts, then inspect the browser tools in app/src/lib/copilot/computer-tools.tsx. Trace how toolsForTurn and HeadlessToolsMiddleware receive computer_* tools for remote chat, and run server/tests/headless-tools.test.ts plus the relevant toolkit tests. Done means a remote AG-UI bot can use caller-offered browser tools while retaining headless behavior and gateway auditing; confirm whether the intended fix is caller pass-through or server-side computer tools.
由索引模型根據 Issue 內容生成。
描述
What happens
In a normal chat with a remote AG-UI Bot (the picked harness, agent-claude-sdk), every browser tool call is refused. The Bot reports:
The tool computer_navigate is unavailable for this headless turn.
The built-in General Assistant browses fine on the same deployment, same page and same request.
Steps to reproduce
Tested at 12c3a69 (main), laptop stack via scripts/start.sh, macOS, Docker (OrbStack).
- Register the picked harness
ghcr.io/copilotkit/openbot-agent-claude-sdk:v0.2.0(PICKED_HARNESS_*set, endpoint allowed viaAGENT_ENDPOINT_ALLOWED_HOSTS). - Open
/channel/new?agent=picked-harness. - Ask:
Open news.ycombinator.com and tell me the top story. - The Bot calls
computer_navigate, gets the refusal above, and falls back to its ownWebFetch.
Doing the same with general-assistant works and is audited as computer.action_allowed.
I only tested agent-claude-sdk. From the code it looks like it affects any remote_ag_ui / remote_mastra Bot (see below), but I haven't confirmed that with the LangGraph Bot.
Why it happens
server/src/copilot.ts:1322buildsclientToolsfrom the Bot's tools that have anexecutefunction.loadToolsForActor(server/src/index.ts:1271) always adds wake-up, memory and host-access tools, so for a remote Bot this list is never empty. That meanscopilot.ts:1445always wraps the run inHeadlessToolsMiddleware.HeadlessToolsMiddleware(toolkit/src/headless-tools.ts:164) treats every pending call whose name starts withcomputer_as its own, whether or not it holds that tool.- In a chat turn the middleware isn't given any computer tools. They're only passed through
toolsForTurnon the routine, approval and hop paths inindex.ts. Socomputer_navigateisn't in its map, and line 184 throws "unavailable for this headless turn". - The
computer_*tools the open page offered ininput.tools(app/src/lib/copilot/computer-tools.tsx) never get to run, even though a browser is open. The middleware's own doc comment says it's for when "the caller has no browser".
Workaround I'm running locally
Leave computer_* calls to the caller when the caller offered that tool:
const executed = new Set<string>();
+ const offeredByCaller = new Set(
+ originalInput.tools
+ .map((tool) => tool.name)
+ .filter((name) => !tools.has(name)),
+ );
...
tools.has(call.function.name) ||
- call.function.name.startsWith("computer_"),
+ (call.function.name.startsWith("computer_") &&
+ !offeredByCaller.has(call.function.name)),
With this change the picked harness browses, and the action still goes through the gateway: it shows up in audit_events as computer.action_allowed with bot: picked-harness and the policy decision attached. toolkit tests pass (59/59), and server/tests/headless-tools.test.ts passes.
This may not be the direction you want, though. If the intent is for remote Bots to run browser tools server-side with no page attached (as the comments in agent-langgraph/src/index.ts suggest), the fix would be to pass createHeadlessComputerTools(...) into the remote chat path the way toolsForTurn does for routines. Either way, refusing the call outright while a page is offering the tool seems unintended.
- 主要語言
- TypeScript
- 星號
- 6.6k
- 分支
- 914
- 平均合併
- 17 小時 52 分鐘
- 30 天內合併 PR
- 155
環境準備
- 提供 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 沒有貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
CopilotKit/OpenBot 的其他 Issue
-
Wrong key location at startup可能已有人在做 @charan-rathore 今天認領。 未關閉
難度 1/5 1 小時以內 新手友好度 68/100
CopilotKit/OpenBot#800 ·
維護者通常 1 天內回覆
-
setup-learning.ts fails on Windows with "spawn npx.cmd EINVAL"可能已有人在做 @Caldas 於 3 天前認領。 未關閉
難度 2/5 1-3 小時 新手友好度 80/100
CopilotKit/OpenBot#792 ·
維護者通常 1 天內回覆
-
Five out-of-subject findings from PR #770's tier-3 review (credential-revoking tests, single-user boot gate, OAuth client rotation, shared-use atomicity, untyped server/tests)可能已有人在做 關聯的 PR 仍在進行中或已合併。 未關閉
難度 4/5 一週以上 新手友好度 22/100
CopilotKit/OpenBot#786 ·
維護者通常 1 天內回覆
-
server/tests leaks database connections: 55 files open a pool and never close it可能已有人在做 @adamsiwiec1 於 3 天前認領。 未關閉
難度 3/5 1-2 天 新手友好度 70/100
CopilotKit/OpenBot#785 ·
維護者通常 1 天內回覆
-
難度 3/5 1-2 天 新手友好度 35/100
CopilotKit/OpenBot#783 ·
維護者通常 1 天內回覆
查看 CopilotKit/OpenBot 的全部 Issue
相似的 Issue
-
難度 1/5 1 小時以內 新手友好度 65/100
521xueweihan/HelloGitHub#4025 ·
-
難度 2/5 1-3 小時 新手友好度 82/100
voidzero-dev/void#478 ·
維護者通常 1 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 69/100
Small-tailqwq/dsh-deep-whale#207 ·
維護者通常 1 天內回覆
-
The inspector never shows the instances field, so scaling out is unreachable可能已有人在做 @Mrkewl 今天認領。 未關閉
難度 2/5 1-3 小時 新手友好度 76/100
xevrion/breakscale#96 ·
維護者通常 4 天內回覆
-
難度 2/5 1-3 小時 新手友好度 70/100
Flux159/mcp-server-kubernetes#384 ·