macOS 27: native app screenshot captures the window-sharing indicator instead of the app window

Aperta
  • Qualcosa non va?
#3,961 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
54/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
macos, swift, typescript
Ambito
desktop, testing-qa

Direzione di ricerca

Start with apple/macos-helper/Sources/AgentDeviceMacOSHelper/BackgroundInteraction.swift and trace onScreenWindows, frontWindow, and the screenshot path through AppWindowScreenshot.frontShareableWindow. Check how accessibility windows can be matched to CGWindow entries, then add regression coverage for screenshots and the frontWindowFrame point check. Done means the app’s main window is selected while the sharing indicator is excluded.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug ready-for-agent

Problem

On macOS 27, agent-device screenshot of a native-backend app session returns the app's window-sharing indicator instead of the app's window. The result is a 132x40 PNG (66x20 points at 2x) of a small "Window" panel.

macOS 27 adds a small layer-0 window to the captured app's own process while anything captures one of that app's windows (here a ScreenCaptureKit stream from another process). The helper's frontWindow takes the first eligible layer-0 window of the pid in CGWindowListCopyWindowInfo order, so the indicator wins over the real window.

Expected: the screenshot captures the app's real main window.

Evidence

  • macOS 27.0 arm64, agent-device 0.21.31, AGENT_DEVICE_MACOS_APP_BACKEND=native, app session on a SwiftUI app (bundle dev.stim.desktop.development.hosted1), driven through a macos-app lease.
  • agent-device snapshot -i lists the indicator before the real window, as an axdialog named "Window" whose only child is a button named "WindowSharingSessionButton" (marked [covered]), then the real window:
@e1 [application] "StimDesktop"
@e2 [axdialog] "Window" [covered]
@e3 [button] "WindowSharingSessionButton" [covered]
@e4 [window] "main"
  • agent-device screenshot out.png then reports out.png (132x40).
  • The app's windows from CGWindowListCopyWindowInfo, filtered to its pid (id, layer, onscreen, width, height, title), from an earlier run with the same symptom:
380 0 1 66 20 Window
264 0 1 1100 850 SwiftUI Playground
  • The indicator survives stopping and relaunching the app while the capture continues.

Cause

onScreenWindows(pid:windowInfo:) in apple/macos-helper/Sources/AgentDeviceMacOSHelper/BackgroundInteraction.swift keeps every window with layer 0, on screen, alpha above 0 and nonzero bounds, and frontWindow returns .first. The 66x20 indicator satisfies all of those. AppWindowScreenshot.frontShareableWindow then captures it, since ScreenCaptureKit offers it.

Fix idea

Choose the screenshot window from the app's accessibility windows instead of the first layer-0 CGWindow: prefer AXMainWindow or AXFocusedWindow, or a window whose subrole is AXStandardWindow, and match it to a CGWindow by frame or _AXUIElementGetWindow. The indicator is an AXDialog without a standard-window subrole, so it is excluded. A size floor would be a weaker fallback.

The same ordering may affect any command that uses frontWindowFrame, such as the on-screen point check for click/press, so a regression test should cover both.

Related: #3213 (native backend follow-ups, capture-frame contract), #3761.

Lingua principale
TypeScript
Stelle
5.2k
Fork
359
Merge medio
13h 23m
PR unite (30g)
655

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di callstack/agent-device

Tutte le issue di callstack/agent-device

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.