macOS 27: native app screenshot captures the window-sharing indicator instead of the app window

オープン
  • 何か問題がありますか？
#3,961 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
54/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
macos, swift, typescript
領域
desktop, testing-qa

調査の方向性

Start with apple/macos-helper/Sources/AgentDeviceMacOSHelper/BackgroundInteraction.swift and trace onScreenWindows, frontWindow, and the screenshot path through AppWindowScreenshot.frontShareableWindow. Check how accessibility windows can be matched to CGWindow entries, then add regression coverage for screenshots and the frontWindowFrame point check. Done means the app’s main window is selected while the sharing indicator is excluded.

索引モデルが issue の本文から書いたものです。

説明

bug ready-for-agent

Problem

On macOS 27, agent-device screenshot of a native-backend app session returns the app's window-sharing indicator instead of the app's window. The result is a 132x40 PNG (66x20 points at 2x) of a small "Window" panel.

macOS 27 adds a small layer-0 window to the captured app's own process while anything captures one of that app's windows (here a ScreenCaptureKit stream from another process). The helper's frontWindow takes the first eligible layer-0 window of the pid in CGWindowListCopyWindowInfo order, so the indicator wins over the real window.

Expected: the screenshot captures the app's real main window.

Evidence

  • macOS 27.0 arm64, agent-device 0.21.31, AGENT_DEVICE_MACOS_APP_BACKEND=native, app session on a SwiftUI app (bundle dev.stim.desktop.development.hosted1), driven through a macos-app lease.
  • agent-device snapshot -i lists the indicator before the real window, as an axdialog named "Window" whose only child is a button named "WindowSharingSessionButton" (marked [covered]), then the real window:
@e1 [application] "StimDesktop"
@e2 [axdialog] "Window" [covered]
@e3 [button] "WindowSharingSessionButton" [covered]
@e4 [window] "main"
  • agent-device screenshot out.png then reports out.png (132x40).
  • The app's windows from CGWindowListCopyWindowInfo, filtered to its pid (id, layer, onscreen, width, height, title), from an earlier run with the same symptom:
380 0 1 66 20 Window
264 0 1 1100 850 SwiftUI Playground
  • The indicator survives stopping and relaunching the app while the capture continues.

Cause

onScreenWindows(pid:windowInfo:) in apple/macos-helper/Sources/AgentDeviceMacOSHelper/BackgroundInteraction.swift keeps every window with layer 0, on screen, alpha above 0 and nonzero bounds, and frontWindow returns .first. The 66x20 indicator satisfies all of those. AppWindowScreenshot.frontShareableWindow then captures it, since ScreenCaptureKit offers it.

Fix idea

Choose the screenshot window from the app's accessibility windows instead of the first layer-0 CGWindow: prefer AXMainWindow or AXFocusedWindow, or a window whose subrole is AXStandardWindow, and match it to a CGWindow by frame or _AXUIElementGetWindow. The indicator is an AXDialog without a standard-window subrole, so it is excluded. A size floor would be a weaker fallback.

The same ordering may affect any command that uses frontWindowFrame, such as the on-screen point check for click/press, so a regression test should cover both.

Related: #3213 (native backend follow-ups, capture-frame contract), #3761.

主要言語
TypeScript
スター
5.3k
フォーク
377
平均マージ
13時間 18分
マージ済み PR（30日）
653

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

callstack/agent-device のほかの issue

callstack/agent-device の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。