Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Remote AG-UI Bots can't use the browser in chat: `computer_*` calls refused as "unavailable for this headless turn"

Aperta
  • Qualcosa non va?
#804 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@charan-rathore ci sta già lavorando.

Dal 18/10/2026.

  • #805 di @charan-rathore — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
68/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
react, typescript
Ambito
ai-infra-agents, backend-api-design

Direzione di ricerca

Start with toolkit/src/headless-tools.ts, server/src/copilot.ts, and server/src/index.ts, then inspect the browser tools in app/src/lib/copilot/computer-tools.tsx. Trace how toolsForTurn and HeadlessToolsMiddleware receive computer_* tools for remote chat, and run server/tests/headless-tools.test.ts plus the relevant toolkit tests. Done means a remote AG-UI bot can use caller-offered browser tools while retaining headless behavior and gateway auditing; confirm whether the intended fix is caller pass-through or server-side computer tools.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

What happens

In a normal chat with a remote AG-UI Bot (the picked harness, agent-claude-sdk), every browser tool call is refused. The Bot reports:

The tool computer_navigate is unavailable for this headless turn.

The built-in General Assistant browses fine on the same deployment, same page and same request.

Steps to reproduce

Tested at 12c3a69 (main), laptop stack via scripts/start.sh, macOS, Docker (OrbStack).

  1. Register the picked harness ghcr.io/copilotkit/openbot-agent-claude-sdk:v0.2.0 (PICKED_HARNESS_* set, endpoint allowed via AGENT_ENDPOINT_ALLOWED_HOSTS).
  2. Open /channel/new?agent=picked-harness.
  3. Ask: Open news.ycombinator.com and tell me the top story.
  4. The Bot calls computer_navigate, gets the refusal above, and falls back to its own WebFetch.

Doing the same with general-assistant works and is audited as computer.action_allowed.

I only tested agent-claude-sdk. From the code it looks like it affects any remote_ag_ui / remote_mastra Bot (see below), but I haven't confirmed that with the LangGraph Bot.

Why it happens
  • server/src/copilot.ts:1322 builds clientTools from the Bot's tools that have an execute function. loadToolsForActor (server/src/index.ts:1271) always adds wake-up, memory and host-access tools, so for a remote Bot this list is never empty. That means copilot.ts:1445 always wraps the run in HeadlessToolsMiddleware.
  • HeadlessToolsMiddleware (toolkit/src/headless-tools.ts:164) treats every pending call whose name starts with computer_ as its own, whether or not it holds that tool.
  • In a chat turn the middleware isn't given any computer tools. They're only passed through toolsForTurn on the routine, approval and hop paths in index.ts. So computer_navigate isn't in its map, and line 184 throws "unavailable for this headless turn".
  • The computer_* tools the open page offered in input.tools (app/src/lib/copilot/computer-tools.tsx) never get to run, even though a browser is open. The middleware's own doc comment says it's for when "the caller has no browser".
Workaround I'm running locally

Leave computer_* calls to the caller when the caller offered that tool:

       const executed = new Set<string>();
+      const offeredByCaller = new Set(
+        originalInput.tools
+          .map((tool) => tool.name)
+          .filter((name) => !tools.has(name)),
+      );
 ...
               tools.has(call.function.name) ||
-              call.function.name.startsWith("computer_"),
+              (call.function.name.startsWith("computer_") &&
+                !offeredByCaller.has(call.function.name)),

With this change the picked harness browses, and the action still goes through the gateway: it shows up in audit_events as computer.action_allowed with bot: picked-harness and the policy decision attached. toolkit tests pass (59/59), and server/tests/headless-tools.test.ts passes.

This may not be the direction you want, though. If the intent is for remote Bots to run browser tools server-side with no page attached (as the comments in agent-langgraph/src/index.ts suggest), the fix would be to pass createHeadlessComputerTools(...) into the remote chat path the way toolsForTurn does for routines. Either way, refusing the call outright while a page is offering the tool seems unintended.

Lingua principale
TypeScript
Stelle
6.6k
Fork
914
Merge medio
17h 52m
PR unite (30g)
155

Preparare l'ambiente

  • Include un Dockerfile o un file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di CopilotKit/OpenBot

Tutte le issue di CopilotKit/OpenBot

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.