Linux support (meta)

Start with the full research report in the issue’s first comment, then compare the build-versus-adopt decision for the proposed Rust host. The client.ts platform gate and the macOS host’s native dependencies identify the shared-layer and host areas affected. Completion means producing the requested go/no-go memo, including adoption, foreground-input, password-fill, and GNOME-extension decisions; this meta issue does not define a standalone implementation task.

Linux support (meta)

Desktop Tools is macOS-only today. This issue tracks what supporting Linux would take and in what order. The full research report (per-area findings, a support matrix for all 48 tools, sources) is in the first comment.

Summary

• What ports and what doesn't. The TypeScript side — the MCP server, schemas, outcomes, snapshots, wait/expect, tables and result bounding — ports with changes. Nothing native does. The host is Swift on a pinned Peekaboo with 52 patches, built on AX, CGEvent, ScreenCaptureKit and TCC, and none of it runs on Linux. client.ts refuses every request when the platform isn't darwin.
• Reading apps works well. AT-SPI2 gives the element tree, actions, values, text, selection and tables on GTK3, Qt 5/6, Firefox and LibreOffice, and mostly on GTK4.
  • Chromium and Electron need accessibility on before they launch.
  • Element screen positions are unreliable on Wayland, so element bounds would come from window-relative extents plus window bounds from the compositor.
• Background input is mostly lost. On Wayland, no standard API delivers pointer or key input to an exact window. On X11 it works only for some toolkits.
  • Semantic actions stay background: AT-SPI DoAction, EditableText and SetCurrentValue.
  • Point clicks, scroll, drag, hover, press and keys become foreground input, through libei/EIS or XTest.
• Wayland has no common window list. Each desktop needs its own backend: a GNOME Shell extension we would ship, KWin scripting, sway/Hyprland/niri IPC, and EWMH on X11.
• Capture works everywhere, by different routes: XComposite on X11; the extension or a one-time ScreenCast portal grant on GNOME; ScreenShot2 on KDE; ext-image-copy-capture on wlroots.
• Linux has no TCC equivalent. X11, the accessibility bus and the Secret Service are open to every process running as the user. The host's value on Linux is its safety contract — exact targets, refusals, caller approval, pause — not gatekeeping a permission. The docs must not claim more than that.
• X11 matters mainly for CI. GNOME 50 dropped its X11 session and Plasma 6.8 is Wayland-only. X11 now matters mainly for Xvfb containers and smaller desktops.

Proposed approach

• Host: a Rust user daemon (zbus, atspi, ashpd, reis, x11rb) behind the same socket and JSON reply shape the macOS client uses. Alongside it:
  • a GTK4 prompt plus tray menu for approval, pause and activity;
  • a small GNOME Shell extension as the compositor helper.
• Backends behind one interface: X11, GNOME, KWin and wlroots.
• Support tiers:

• Tool coverage: about 35 of 48 tools map cleanly or with degradations.
  • The pointer tools and desktop_key become foreground-only on Wayland.
  • desktop_dock has no equivalent.
  • App hide/unhide would be emulated by minimizing windows.
  • desktop_switch_space works better than on macOS: it's a direct API call, not a keyboard shortcut.

Decide first

• Build or adopt. Evaluate agent-sh/computer-use-linux and the Cua Driver, both MIT Rust and both already covering much of this. Could either become the Linux host behind our protocol and contract?
• Foreground input by default? On Wayland it moves the real pointer and focus while the user may be working. Should Linux default to semantic actions only, with foreground input as an explicit opt-in?
• Password fill. The Secret Service has no per-item prompt. Options: require approval for every fill, support only KeePassXC (which confirms each access), or leave fill out on Linux.
• GNOME extension. Can we ship and maintain it across GNOME releases? Without it, GNOME support drops to portal prompts and no window list.

Work (one sub-issue each)

Phase 0: spike (2–3 weeks)

• Spike: AT-SPI tree, window matching, capture, DoAction and an EIS click on Xvfb and GNOME Wayland, ending in a go/no-go memo
• Build-versus-adopt evaluation

Phase 1: make the shared layer platform-neutral (2–3 weeks; macOS behavior unchanged)

• Replace the darwin gate with backend selection
• Add a capabilities request, and filter tools/list by backend
• Platform-neutral role, action and app-ID vocabulary (bundle_id → app_id, the AX action enum)
• Write down the host contract: receipts, refusal codes, pause, activity log, overlay
• Platform-specific tool descriptions and server instructions

Phase 2: Linux host on X11/Xvfb (8–10 weeks)

• Host skeleton: socket, caller approval via peer credentials, pause, activity log, self-target refusal
• Apps, windows, inspect, read, wait and screen via AT-SPI, EWMH and XComposite
• Semantic actions: click on element, type, set value, select, element action, insert text
• Foreground input via XTest, and XSendEvent where it works
• Clipboard, launch/open/quit, window operations, workspaces, OCR
• Headless CI on Xvfb with GTK3, GTK4, Qt 6, Firefox, Chromium and LibreOffice fixtures

Phase 3: GNOME Wayland (8–10 weeks)

• GNOME Shell extension: window list, focus, geometry, stacking, capture, workspaces, notifications, input sources, overlay, caller check
• RemoteDesktop portal + EIS for foreground input, with token storage and re-prompting
• systemd user service, tray, and approval UI

Phase 4: KDE Plasma 6 Wayland (4–6 weeks)

• KWin scripting backend, ScreenShot2, EIS, clipboard, input sources, overlay

Phase 5: wlroots (4–6 weeks)

• sway, Hyprland and niri backends; screen capture; virtual input; notification daemons

Phase 6: release (4–6 weeks)

• Packaging as deb, rpm and AppImage (not Flatpak, which can't act as assistive technology)
• Linux docs: setup per desktop, a support matrix, the threat model, and a doctor command
• Release workflow

Rough total: 6–9 engineer-months for phases 0–4, a beta on X11, GNOME and KDE. Phases 5–6 add 2–3 months. Adopting an existing host could cut phases 2–4 by about a third.

Risks

• Exact targeting on Wayland. Matching AT-SPI windows to compositor windows is heuristic. Same-titled windows in one process, such as browser or terminal windows, may be refused.
• Portal tokens are single-use and rotate, and a locked session can invalidate them.
• GTK4 accessibility gaps affect GNOME's own apps: screen extents, tables and some actions.
• The test matrix is broad: GNOME 49–51, Plasma 6.x, sway, Hyprland, niri and Xvfb, each with a headless CI session.