macOS 27: native app screenshot captures the window-sharing indicator instead of the app window

Aberta
  • Algo errado?
#3,961 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
54/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
macos, swift, typescript
Domínio
desktop, testing-qa

Direção de pesquisa

Start with apple/macos-helper/Sources/AgentDeviceMacOSHelper/BackgroundInteraction.swift and trace onScreenWindows, frontWindow, and the screenshot path through AppWindowScreenshot.frontShareableWindow. Check how accessibility windows can be matched to CGWindow entries, then add regression coverage for screenshots and the frontWindowFrame point check. Done means the app’s main window is selected while the sharing indicator is excluded.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

bug ready-for-agent

Problem

On macOS 27, agent-device screenshot of a native-backend app session returns the app's window-sharing indicator instead of the app's window. The result is a 132x40 PNG (66x20 points at 2x) of a small "Window" panel.

macOS 27 adds a small layer-0 window to the captured app's own process while anything captures one of that app's windows (here a ScreenCaptureKit stream from another process). The helper's frontWindow takes the first eligible layer-0 window of the pid in CGWindowListCopyWindowInfo order, so the indicator wins over the real window.

Expected: the screenshot captures the app's real main window.

Evidence

  • macOS 27.0 arm64, agent-device 0.21.31, AGENT_DEVICE_MACOS_APP_BACKEND=native, app session on a SwiftUI app (bundle dev.stim.desktop.development.hosted1), driven through a macos-app lease.
  • agent-device snapshot -i lists the indicator before the real window, as an axdialog named "Window" whose only child is a button named "WindowSharingSessionButton" (marked [covered]), then the real window:
@e1 [application] "StimDesktop"
@e2 [axdialog] "Window" [covered]
@e3 [button] "WindowSharingSessionButton" [covered]
@e4 [window] "main"
  • agent-device screenshot out.png then reports out.png (132x40).
  • The app's windows from CGWindowListCopyWindowInfo, filtered to its pid (id, layer, onscreen, width, height, title), from an earlier run with the same symptom:
380 0 1 66 20 Window
264 0 1 1100 850 SwiftUI Playground
  • The indicator survives stopping and relaunching the app while the capture continues.

Cause

onScreenWindows(pid:windowInfo:) in apple/macos-helper/Sources/AgentDeviceMacOSHelper/BackgroundInteraction.swift keeps every window with layer 0, on screen, alpha above 0 and nonzero bounds, and frontWindow returns .first. The 66x20 indicator satisfies all of those. AppWindowScreenshot.frontShareableWindow then captures it, since ScreenCaptureKit offers it.

Fix idea

Choose the screenshot window from the app's accessibility windows instead of the first layer-0 CGWindow: prefer AXMainWindow or AXFocusedWindow, or a window whose subrole is AXStandardWindow, and match it to a CGWindow by frame or _AXUIElementGetWindow. The indicator is an AXDialog without a standard-window subrole, so it is excluded. A size floor would be a weaker fallback.

The same ordering may affect any command that uses frontWindowFrame, such as the on-screen point check for click/press, so a regression test should cover both.

Related: #3213 (native backend follow-ups, capture-frame contract), #3761.

Linguagem predominante
TypeScript
Estrelas
5.2k
Forks
359
Merge médio
13h 37min
PRs com merge (30d)
665

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de callstack/agent-device

Todas as issues de callstack/agent-device

Issues semelhantes

Mais issues de TypeScript

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.