Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

macOS: drive a local app without activating it or moving the real pointer

Đang mở
  • Có gì không ổn?
#3,254 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
35/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
swift, typescript
Lĩnh vực
desktop, testing

Hướng nghiên cứu

Start with buildMacOpenArgs in packages/platform-apple/src/os/macos/host-provider.ts, then read RunnerTests+Lifecycle.swift, RunnerTests+CommandExecution.swift, and MouseClickDelivery.swift for the activation and click paths. Check the related #3213 item 13 for window-field context. Done means the proposed opt-in foreground behavior, background interaction and app-window screenshot behavior are implemented and tested, with the remaining activation limits documented.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Problem

Driving a local macOS app with agent-device takes the app to the front and moves the user's real pointer, even when the app could be driven in the background. For an agent running on the user's own Mac this interrupts whatever the user is doing in another app. The opt-in native backend (AGENT_DEVICE_MACOS_APP_BACKEND=native, ADR 0031, #3213) already acts through accessibility actions without XCTest and without the "Automation Running" overlay, but open, the XCTest runner and the coordinate click path still bring the target app forward or use real pointer events. This issue is about those remaining activation points. It is related to #3213 (item 13, pointer events with window fields) and #3106 (non-activating observations), and does not replace either.

Measured behavior

Fixture: a small AppKit app launched in the background (it logs NSApp.isActive itself and the test records NSWorkspace.frontmostApplication). agent-device 0.21.12, with 0.21.20 checked for the points below.

  • agent-device open <bundleId> --platform macos activates the app: the fixture logged didBecomeActive active=true key=true and frontmostApplication changed from the previously front app to the fixture. The macOS open path runs open -b <bundleId> (buildMacOpenArgs in packages/platform-apple/src/os/macos/host-provider.ts), without -g.
  • Without the native backend, every macOS command goes through the XCTest runner, which shows the "Automation Running" overlay.
  • Intermittent AXError -25204 (cannotComplete) from AXUIElementCopyElementAtPosition on the first call after the app has been idle in the background. A retry succeeds.

Read from the source on main, not measured:

  • Runner: RunnerTests+Lifecycle.swift calls target.activate() unless the target is already .runningForeground, so each runner command on an inactive app activates it. screenshot with an app bundle id calls targetApp.activate() in RunnerTests+CommandExecution.swift.
  • Helper: MouseClickDelivery.swift posts clicks with CGEvent.post(tap: .cghidEventTap) at screen coordinates. That moves the real pointer, lands on whatever window is topmost at that point, and the system activates the clicked app. It is not delivered to the target process.

Related measurement from #3213: CGEventPostToPid with the window number set delivers clicks to a background app, while a bare post delivers nothing.

Proposal

  1. open: do not activate the app unless --foreground is passed (open -g, or NSWorkspace.OpenConfiguration.activates = false). An app that is already running is attached to, not raised. --foreground keeps today's behavior.
  2. Native backend: keep acting through accessibility actions (AXPress, AXSetValue, scroll) addressed by element ref. For raw coordinates and keys, use CGEvent.postToPid with the window fields from #3213 item 13 instead of .cghidEventTap. In my fixtures, postToPid with the window set delivered clicks on buttons, text fields and table rows, text, Backspace and Tab, Command shortcuts and scroll while the app was inactive. Clicks on views that reject the first mouse (custom views, SwiftUI onTapGesture) were dropped, so those need an explicit activation step (--foreground, or a focus command).
  3. XCTest runner on macOS: skip target.activate() unless --foreground.
  4. screenshot of an app: capture the target window (ScreenCaptureKit window capture) instead of activating the app first.
  5. Document the native backend variable and which commands can still activate the app.

Not in scope

Drags and WKWebView button clicks, which #3213 measured as needing the private focus-without-raise call.

Ngôn ngữ chính
TypeScript
Star
4.9k
Fork
328
Merge trung bình
11 giờ 51 phút
Pull request đã merge (30 ngày)
535

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của callstack/agent-device

Tất cả issue của callstack/agent-device

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.