macOS 27: native app screenshot captures the window-sharing indicator instead of the app window

未关闭
  • 哪里不对？
#3,961 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
54/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
macos, swift, typescript
领域
desktop, testing-qa

调研方向

Start with apple/macos-helper/Sources/AgentDeviceMacOSHelper/BackgroundInteraction.swift and trace onScreenWindows, frontWindow, and the screenshot path through AppWindowScreenshot.frontShareableWindow. Check how accessibility windows can be matched to CGWindow entries, then add regression coverage for screenshots and the frontWindowFrame point check. Done means the app’s main window is selected while the sharing indicator is excluded.

由索引模型根据 Issue 内容生成。

描述

bug ready-for-agent

Problem

On macOS 27, agent-device screenshot of a native-backend app session returns the app's window-sharing indicator instead of the app's window. The result is a 132x40 PNG (66x20 points at 2x) of a small "Window" panel.

macOS 27 adds a small layer-0 window to the captured app's own process while anything captures one of that app's windows (here a ScreenCaptureKit stream from another process). The helper's frontWindow takes the first eligible layer-0 window of the pid in CGWindowListCopyWindowInfo order, so the indicator wins over the real window.

Expected: the screenshot captures the app's real main window.

Evidence

  • macOS 27.0 arm64, agent-device 0.21.31, AGENT_DEVICE_MACOS_APP_BACKEND=native, app session on a SwiftUI app (bundle dev.stim.desktop.development.hosted1), driven through a macos-app lease.
  • agent-device snapshot -i lists the indicator before the real window, as an axdialog named "Window" whose only child is a button named "WindowSharingSessionButton" (marked [covered]), then the real window:
@e1 [application] "StimDesktop"
@e2 [axdialog] "Window" [covered]
@e3 [button] "WindowSharingSessionButton" [covered]
@e4 [window] "main"
  • agent-device screenshot out.png then reports out.png (132x40).
  • The app's windows from CGWindowListCopyWindowInfo, filtered to its pid (id, layer, onscreen, width, height, title), from an earlier run with the same symptom:
380 0 1 66 20 Window
264 0 1 1100 850 SwiftUI Playground
  • The indicator survives stopping and relaunching the app while the capture continues.

Cause

onScreenWindows(pid:windowInfo:) in apple/macos-helper/Sources/AgentDeviceMacOSHelper/BackgroundInteraction.swift keeps every window with layer 0, on screen, alpha above 0 and nonzero bounds, and frontWindow returns .first. The 66x20 indicator satisfies all of those. AppWindowScreenshot.frontShareableWindow then captures it, since ScreenCaptureKit offers it.

Fix idea

Choose the screenshot window from the app's accessibility windows instead of the first layer-0 CGWindow: prefer AXMainWindow or AXFocusedWindow, or a window whose subrole is AXStandardWindow, and match it to a CGWindow by frame or _AXUIElementGetWindow. The indicator is an AXDialog without a standard-window subrole, so it is excluded. A size floor would be a weaker fallback.

The same ordering may affect any command that uses frontWindowFrame, such as the on-screen point check for click/press, so a regression test should cover both.

Related: #3213 (native backend follow-ups, capture-frame contract), #3761.

主要语言
TypeScript
星标
5.2k
派生
359
平均合并
13 小时 31 分钟
30 天内合并 PR
664

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

callstack/agent-device 的其他 Issue

查看 callstack/agent-device 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。