macOS: drive a local app without activating it or moving the real pointer
メンテナーはふだん 1 日以内に返信
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 35/100
- issue の種類
- 機能追加
- 明瞭さ
- おおむね明確
- 活発さ
- 活発
- 技術スタック
- swift, typescript
調査の方向性
Start with buildMacOpenArgs in packages/platform-apple/src/os/macos/host-provider.ts, then read RunnerTests+Lifecycle.swift, RunnerTests+CommandExecution.swift, and MouseClickDelivery.swift for the activation and click paths. Check the related #3213 item 13 for window-field context. Done means the proposed opt-in foreground behavior, background interaction and app-window screenshot behavior are implemented and tested, with the remaining activation limits documented.
索引モデルが issue の本文から書いたものです。
説明
Problem
Driving a local macOS app with agent-device takes the app to the front and moves the user's real pointer, even when the app could be driven in the background. For an agent running on the user's own Mac this interrupts whatever the user is doing in another app. The opt-in native backend (AGENT_DEVICE_MACOS_APP_BACKEND=native, ADR 0031, #3213) already acts through accessibility actions without XCTest and without the "Automation Running" overlay, but open, the XCTest runner and the coordinate click path still bring the target app forward or use real pointer events. This issue is about those remaining activation points. It is related to #3213 (item 13, pointer events with window fields) and #3106 (non-activating observations), and does not replace either.
Measured behavior
Fixture: a small AppKit app launched in the background (it logs NSApp.isActive itself and the test records NSWorkspace.frontmostApplication). agent-device 0.21.12, with 0.21.20 checked for the points below.
agent-device open <bundleId> --platform macosactivates the app: the fixture loggeddidBecomeActive active=true key=trueandfrontmostApplicationchanged from the previously front app to the fixture. The macOS open path runsopen -b <bundleId>(buildMacOpenArgsinpackages/platform-apple/src/os/macos/host-provider.ts), without-g.- Without the native backend, every macOS command goes through the XCTest runner, which shows the "Automation Running" overlay.
- Intermittent
AXError -25204(cannotComplete) fromAXUIElementCopyElementAtPositionon the first call after the app has been idle in the background. A retry succeeds.
Read from the source on main, not measured:
- Runner:
RunnerTests+Lifecycle.swiftcallstarget.activate()unless the target is already.runningForeground, so each runner command on an inactive app activates it.screenshotwith an app bundle id callstargetApp.activate()inRunnerTests+CommandExecution.swift. - Helper:
MouseClickDelivery.swiftposts clicks withCGEvent.post(tap: .cghidEventTap)at screen coordinates. That moves the real pointer, lands on whatever window is topmost at that point, and the system activates the clicked app. It is not delivered to the target process.
Related measurement from #3213: CGEventPostToPid with the window number set delivers clicks to a background app, while a bare post delivers nothing.
Proposal
open: do not activate the app unless--foregroundis passed (open -g, orNSWorkspace.OpenConfiguration.activates = false). An app that is already running is attached to, not raised.--foregroundkeeps today's behavior.- Native backend: keep acting through accessibility actions (
AXPress,AXSetValue, scroll) addressed by element ref. For raw coordinates and keys, useCGEvent.postToPidwith the window fields from #3213 item 13 instead of.cghidEventTap. In my fixtures,postToPidwith the window set delivered clicks on buttons, text fields and table rows, text, Backspace and Tab, Command shortcuts and scroll while the app was inactive. Clicks on views that reject the first mouse (custom views, SwiftUIonTapGesture) were dropped, so those need an explicit activation step (--foreground, or afocuscommand). - XCTest runner on macOS: skip
target.activate()unless--foreground. screenshotof an app: capture the target window (ScreenCaptureKit window capture) instead of activating the app first.- Document the native backend variable and which commands can still activate the app.
Not in scope
Drags and WKWebView button clicks, which #3213 measured as needing the private focus-without-raise call.
- 主要言語
- TypeScript
- スター
- 4.9k
- フォーク
- 328
- 平均マージ
- 12時間 13分
- マージ済み PR（30日）
- 541
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
callstack/agent-device のほかの issue
-
settings clear-app-state fails with ENOENT … scandir '(null)' for iOS system apps such as Settings対応中かも @thymikee が今日担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 80/100
callstack/agent-device#3305 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
callstack/agent-device#1869 ·
メンテナーはふだん 1 日以内に返信
-
install and reinstall from another session in the same daemon replace the app on a claimed deviceオープン
難易度 4/5 3〜5日 初心者へのやさしさ 55/100
callstack/agent-device#3345 ·
メンテナーはふだん 1 日以内に返信
-
難易度 4/5 3〜5日 初心者へのやさしさ 48/100
callstack/agent-device#3344 ·
メンテナーはふだん 1 日以内に返信
-
難易度 5/5 1週間以上 初心者へのやさしさ 35/100
callstack/agent-device#3342 · コメント 7 件 ·
メンテナーはふだん 1 日以内に返信
callstack/agent-device の issue をすべて見る
似ている issue
-
Flaky: mongodb-memory-server 'Port already in use' when another process starts a mongod concurrentlyオープンarea:testing bug effort:S priority:P2
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
メンテナーはふだん 1 日以内に返信
-
lens:agent lens:process process
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
thebristolsound/birdbrain#1772 ·
メンテナーはふだん 1 日以内に返信
-
bug priority:low ready-for-dev
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
Automattic/data-liberation-agent#685 ·
メンテナーはふだん 1 日以内に返信
-
Business
難易度 2/5 1〜3時間 初心者へのやさしさ 66/100
メンテナーはふだん 1 日以内に返信