macOS 27: native app screenshot captures the window-sharing indicator instead of the app window

Ouverte
  • Quelque chose ne va pas ?
#3,961 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
54/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
macos, swift, typescript
Domaine
desktop, testing-qa

Piste de recherche

Start with apple/macos-helper/Sources/AgentDeviceMacOSHelper/BackgroundInteraction.swift and trace onScreenWindows, frontWindow, and the screenshot path through AppWindowScreenshot.frontShareableWindow. Check how accessibility windows can be matched to CGWindow entries, then add regression coverage for screenshots and the frontWindowFrame point check. Done means the app’s main window is selected while the sharing indicator is excluded.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

bug ready-for-agent

Problem

On macOS 27, agent-device screenshot of a native-backend app session returns the app's window-sharing indicator instead of the app's window. The result is a 132x40 PNG (66x20 points at 2x) of a small "Window" panel.

macOS 27 adds a small layer-0 window to the captured app's own process while anything captures one of that app's windows (here a ScreenCaptureKit stream from another process). The helper's frontWindow takes the first eligible layer-0 window of the pid in CGWindowListCopyWindowInfo order, so the indicator wins over the real window.

Expected: the screenshot captures the app's real main window.

Evidence

  • macOS 27.0 arm64, agent-device 0.21.31, AGENT_DEVICE_MACOS_APP_BACKEND=native, app session on a SwiftUI app (bundle dev.stim.desktop.development.hosted1), driven through a macos-app lease.
  • agent-device snapshot -i lists the indicator before the real window, as an axdialog named "Window" whose only child is a button named "WindowSharingSessionButton" (marked [covered]), then the real window:
@e1 [application] "StimDesktop"
@e2 [axdialog] "Window" [covered]
@e3 [button] "WindowSharingSessionButton" [covered]
@e4 [window] "main"
  • agent-device screenshot out.png then reports out.png (132x40).
  • The app's windows from CGWindowListCopyWindowInfo, filtered to its pid (id, layer, onscreen, width, height, title), from an earlier run with the same symptom:
380 0 1 66 20 Window
264 0 1 1100 850 SwiftUI Playground
  • The indicator survives stopping and relaunching the app while the capture continues.

Cause

onScreenWindows(pid:windowInfo:) in apple/macos-helper/Sources/AgentDeviceMacOSHelper/BackgroundInteraction.swift keeps every window with layer 0, on screen, alpha above 0 and nonzero bounds, and frontWindow returns .first. The 66x20 indicator satisfies all of those. AppWindowScreenshot.frontShareableWindow then captures it, since ScreenCaptureKit offers it.

Fix idea

Choose the screenshot window from the app's accessibility windows instead of the first layer-0 CGWindow: prefer AXMainWindow or AXFocusedWindow, or a window whose subrole is AXStandardWindow, and match it to a CGWindow by frame or _AXUIElementGetWindow. The indicator is an AXDialog without a standard-window subrole, so it is excluded. A size floor would be a weaker fallback.

The same ordering may affect any command that uses frontWindowFrame, such as the on-screen point check for click/press, so a regression test should cover both.

Related: #3213 (native backend follow-ups, capture-frame contract), #3761.

Langage dominant
TypeScript
Étoiles
5.2k
Forks
359
Merge moyen
13 h 23 min
PR mergées (30 j)
655

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de callstack/agent-device

Toutes les issues de callstack/agent-device

Issues similaires

Plus d'issues TypeScript

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.