Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

Remote AG-UI Bots can't use the browser in chat: `computer_*` calls refused as "unavailable for this headless turn"

Offen
  • Stimmt etwas nicht?
#804 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@charan-rathore arbeitet bereits daran.

Seit 18.10.2026.

  • #805 von @charan-rathore — offen

Bewertung

Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Anfängerfreundlichkeit
68/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
react, typescript
Bereich
ai-infra-agents, backend-api-design

Rechercherichtung

Start with toolkit/src/headless-tools.ts, server/src/copilot.ts, and server/src/index.ts, then inspect the browser tools in app/src/lib/copilot/computer-tools.tsx. Trace how toolsForTurn and HeadlessToolsMiddleware receive computer_* tools for remote chat, and run server/tests/headless-tools.test.ts plus the relevant toolkit tests. Done means a remote AG-UI bot can use caller-offered browser tools while retaining headless behavior and gateway auditing; confirm whether the intended fix is caller pass-through or server-side computer tools.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

What happens

In a normal chat with a remote AG-UI Bot (the picked harness, agent-claude-sdk), every browser tool call is refused. The Bot reports:

The tool computer_navigate is unavailable for this headless turn.

The built-in General Assistant browses fine on the same deployment, same page and same request.

Steps to reproduce

Tested at 12c3a69 (main), laptop stack via scripts/start.sh, macOS, Docker (OrbStack).

  1. Register the picked harness ghcr.io/copilotkit/openbot-agent-claude-sdk:v0.2.0 (PICKED_HARNESS_* set, endpoint allowed via AGENT_ENDPOINT_ALLOWED_HOSTS).
  2. Open /channel/new?agent=picked-harness.
  3. Ask: Open news.ycombinator.com and tell me the top story.
  4. The Bot calls computer_navigate, gets the refusal above, and falls back to its own WebFetch.

Doing the same with general-assistant works and is audited as computer.action_allowed.

I only tested agent-claude-sdk. From the code it looks like it affects any remote_ag_ui / remote_mastra Bot (see below), but I haven't confirmed that with the LangGraph Bot.

Why it happens
  • server/src/copilot.ts:1322 builds clientTools from the Bot's tools that have an execute function. loadToolsForActor (server/src/index.ts:1271) always adds wake-up, memory and host-access tools, so for a remote Bot this list is never empty. That means copilot.ts:1445 always wraps the run in HeadlessToolsMiddleware.
  • HeadlessToolsMiddleware (toolkit/src/headless-tools.ts:164) treats every pending call whose name starts with computer_ as its own, whether or not it holds that tool.
  • In a chat turn the middleware isn't given any computer tools. They're only passed through toolsForTurn on the routine, approval and hop paths in index.ts. So computer_navigate isn't in its map, and line 184 throws "unavailable for this headless turn".
  • The computer_* tools the open page offered in input.tools (app/src/lib/copilot/computer-tools.tsx) never get to run, even though a browser is open. The middleware's own doc comment says it's for when "the caller has no browser".
Workaround I'm running locally

Leave computer_* calls to the caller when the caller offered that tool:

       const executed = new Set<string>();
+      const offeredByCaller = new Set(
+        originalInput.tools
+          .map((tool) => tool.name)
+          .filter((name) => !tools.has(name)),
+      );
 ...
               tools.has(call.function.name) ||
-              call.function.name.startsWith("computer_"),
+              (call.function.name.startsWith("computer_") &&
+                !offeredByCaller.has(call.function.name)),

With this change the picked harness browses, and the action still goes through the gateway: it shows up in audit_events as computer.action_allowed with bot: picked-harness and the policy decision attached. toolkit tests pass (59/59), and server/tests/headless-tools.test.ts passes.

This may not be the direction you want, though. If the intent is for remote Bots to run browser tools server-side with no page attached (as the comments in agent-langgraph/src/index.ts suggest), the fix would be to pass createHeadlessComputerTools(...) into the remote chat path the way toolsForTurn does for routines. Either way, refusing the call outright while a page is offering the tool seems unintended.

Vorherrschende Sprache
TypeScript
Sterne
6.6k
Forks
914
Ø Merge
19 Std. 22 Min.
Gemergte PRs (30 T.)
140

Entwicklungsumgebung

  • Enthält ein Dockerfile oder eine Docker-Compose-Datei
  • Hat eine Pull-Request-Vorlage
  • Kein Beitragsleitfaden

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus CopilotKit/OpenBot

Alle Issues in CopilotKit/OpenBot

Ähnliche Issues

Weitere Issues zu TypeScript

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.