Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Remote AG-UI Bots can't use the browser in chat: `computer_*` calls refused as "unavailable for this headless turn"

Ouverte
  • Quelque chose ne va pas ?
#804 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@charan-rathore y travaille déjà.

Depuis le 18/10/2026.

  • #805 par @charan-rathore — ouverte

Évaluation

Difficulté
3/5
Temps estimé
1-2 jours
Accessibilité débutants
68/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
react, typescript
Domaine
ai-infra-agents, backend-api-design

Piste de recherche

Start with toolkit/src/headless-tools.ts, server/src/copilot.ts, and server/src/index.ts, then inspect the browser tools in app/src/lib/copilot/computer-tools.tsx. Trace how toolsForTurn and HeadlessToolsMiddleware receive computer_* tools for remote chat, and run server/tests/headless-tools.test.ts plus the relevant toolkit tests. Done means a remote AG-UI bot can use caller-offered browser tools while retaining headless behavior and gateway auditing; confirm whether the intended fix is caller pass-through or server-side computer tools.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

What happens

In a normal chat with a remote AG-UI Bot (the picked harness, agent-claude-sdk), every browser tool call is refused. The Bot reports:

The tool computer_navigate is unavailable for this headless turn.

The built-in General Assistant browses fine on the same deployment, same page and same request.

Steps to reproduce

Tested at 12c3a69 (main), laptop stack via scripts/start.sh, macOS, Docker (OrbStack).

  1. Register the picked harness ghcr.io/copilotkit/openbot-agent-claude-sdk:v0.2.0 (PICKED_HARNESS_* set, endpoint allowed via AGENT_ENDPOINT_ALLOWED_HOSTS).
  2. Open /channel/new?agent=picked-harness.
  3. Ask: Open news.ycombinator.com and tell me the top story.
  4. The Bot calls computer_navigate, gets the refusal above, and falls back to its own WebFetch.

Doing the same with general-assistant works and is audited as computer.action_allowed.

I only tested agent-claude-sdk. From the code it looks like it affects any remote_ag_ui / remote_mastra Bot (see below), but I haven't confirmed that with the LangGraph Bot.

Why it happens
  • server/src/copilot.ts:1322 builds clientTools from the Bot's tools that have an execute function. loadToolsForActor (server/src/index.ts:1271) always adds wake-up, memory and host-access tools, so for a remote Bot this list is never empty. That means copilot.ts:1445 always wraps the run in HeadlessToolsMiddleware.
  • HeadlessToolsMiddleware (toolkit/src/headless-tools.ts:164) treats every pending call whose name starts with computer_ as its own, whether or not it holds that tool.
  • In a chat turn the middleware isn't given any computer tools. They're only passed through toolsForTurn on the routine, approval and hop paths in index.ts. So computer_navigate isn't in its map, and line 184 throws "unavailable for this headless turn".
  • The computer_* tools the open page offered in input.tools (app/src/lib/copilot/computer-tools.tsx) never get to run, even though a browser is open. The middleware's own doc comment says it's for when "the caller has no browser".
Workaround I'm running locally

Leave computer_* calls to the caller when the caller offered that tool:

       const executed = new Set<string>();
+      const offeredByCaller = new Set(
+        originalInput.tools
+          .map((tool) => tool.name)
+          .filter((name) => !tools.has(name)),
+      );
 ...
               tools.has(call.function.name) ||
-              call.function.name.startsWith("computer_"),
+              (call.function.name.startsWith("computer_") &&
+                !offeredByCaller.has(call.function.name)),

With this change the picked harness browses, and the action still goes through the gateway: it shows up in audit_events as computer.action_allowed with bot: picked-harness and the policy decision attached. toolkit tests pass (59/59), and server/tests/headless-tools.test.ts passes.

This may not be the direction you want, though. If the intent is for remote Bots to run browser tools server-side with no page attached (as the comments in agent-langgraph/src/index.ts suggest), the fix would be to pass createHeadlessComputerTools(...) into the remote chat path the way toolsForTurn does for routines. Either way, refusing the call outright while a page is offering the tool seems unintended.

Langage dominant
TypeScript
Étoiles
6.6k
Forks
914
Merge moyen
17 h 52 min
PR mergées (30 j)
155

Préparer son environnement

  • Fournit un Dockerfile ou un fichier Docker Compose
  • Propose un modèle de pull request
  • Aucun guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de CopilotKit/OpenBot

Toutes les issues de CopilotKit/OpenBot

Issues similaires

Plus d'issues TypeScript

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.