Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Remote AG-UI Bots can't use the browser in chat: `computer_*` calls refused as "unavailable for this headless turn"

Aberta
  • Algo errado?
#804 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

@charan-rathore já está trabalhando nisso.

Desde 18/10/2026.

  • #805 de @charan-rathore — aberto

Avaliação

Dificuldade
3/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
68/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
react, typescript
Domínio
ai-infra-agents, backend-api-design

Direção de pesquisa

Start with toolkit/src/headless-tools.ts, server/src/copilot.ts, and server/src/index.ts, then inspect the browser tools in app/src/lib/copilot/computer-tools.tsx. Trace how toolsForTurn and HeadlessToolsMiddleware receive computer_* tools for remote chat, and run server/tests/headless-tools.test.ts plus the relevant toolkit tests. Done means a remote AG-UI bot can use caller-offered browser tools while retaining headless behavior and gateway auditing; confirm whether the intended fix is caller pass-through or server-side computer tools.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

What happens

In a normal chat with a remote AG-UI Bot (the picked harness, agent-claude-sdk), every browser tool call is refused. The Bot reports:

The tool computer_navigate is unavailable for this headless turn.

The built-in General Assistant browses fine on the same deployment, same page and same request.

Steps to reproduce

Tested at 12c3a69 (main), laptop stack via scripts/start.sh, macOS, Docker (OrbStack).

  1. Register the picked harness ghcr.io/copilotkit/openbot-agent-claude-sdk:v0.2.0 (PICKED_HARNESS_* set, endpoint allowed via AGENT_ENDPOINT_ALLOWED_HOSTS).
  2. Open /channel/new?agent=picked-harness.
  3. Ask: Open news.ycombinator.com and tell me the top story.
  4. The Bot calls computer_navigate, gets the refusal above, and falls back to its own WebFetch.

Doing the same with general-assistant works and is audited as computer.action_allowed.

I only tested agent-claude-sdk. From the code it looks like it affects any remote_ag_ui / remote_mastra Bot (see below), but I haven't confirmed that with the LangGraph Bot.

Why it happens
  • server/src/copilot.ts:1322 builds clientTools from the Bot's tools that have an execute function. loadToolsForActor (server/src/index.ts:1271) always adds wake-up, memory and host-access tools, so for a remote Bot this list is never empty. That means copilot.ts:1445 always wraps the run in HeadlessToolsMiddleware.
  • HeadlessToolsMiddleware (toolkit/src/headless-tools.ts:164) treats every pending call whose name starts with computer_ as its own, whether or not it holds that tool.
  • In a chat turn the middleware isn't given any computer tools. They're only passed through toolsForTurn on the routine, approval and hop paths in index.ts. So computer_navigate isn't in its map, and line 184 throws "unavailable for this headless turn".
  • The computer_* tools the open page offered in input.tools (app/src/lib/copilot/computer-tools.tsx) never get to run, even though a browser is open. The middleware's own doc comment says it's for when "the caller has no browser".
Workaround I'm running locally

Leave computer_* calls to the caller when the caller offered that tool:

       const executed = new Set<string>();
+      const offeredByCaller = new Set(
+        originalInput.tools
+          .map((tool) => tool.name)
+          .filter((name) => !tools.has(name)),
+      );
 ...
               tools.has(call.function.name) ||
-              call.function.name.startsWith("computer_"),
+              (call.function.name.startsWith("computer_") &&
+                !offeredByCaller.has(call.function.name)),

With this change the picked harness browses, and the action still goes through the gateway: it shows up in audit_events as computer.action_allowed with bot: picked-harness and the policy decision attached. toolkit tests pass (59/59), and server/tests/headless-tools.test.ts passes.

This may not be the direction you want, though. If the intent is for remote Bots to run browser tools server-side with no page attached (as the comments in agent-langgraph/src/index.ts suggest), the fix would be to pass createHeadlessComputerTools(...) into the remote chat path the way toolsForTurn does for routines. Either way, refusing the call outright while a page is offering the tool seems unintended.

Linguagem predominante
TypeScript
Estrelas
6.6k
Forks
914
Merge médio
22h 56min
PRs com merge (30d)
141

Preparar o ambiente

  • Inclui um Dockerfile ou arquivo Docker Compose
  • Tem um modelo de pull request
  • Sem guia de contribuição

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de CopilotKit/OpenBot

Todas as issues de CopilotKit/OpenBot

Issues semelhantes

Mais issues de TypeScript

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.