iOS: XCTest runner publishes window-relative frames and shifted taps for an iPad app that does not fill the screen (Windowed Apps / Stage Manager)

Start with RunnerTests+SnapshotAcquisition.swift and RunnerTests+Interaction.swift, then read ADR 0004 and the snapshot geometry code in the Swift and TypeScript packages. Reproduce on an iPad Simulator with Windowed Apps enabled using the provided Calendar commands, then check viewport, rects, taps, and focused text input against the on-screen window. The issue’s completion conditions define the required results; an open linked pull request already addresses it.

Purpose

On an iPad Simulator with Windowed Apps on, an app window that does not fill the screen gets wrong geometry from the XCTest runner. Rects, the viewport, and every coordinate action disagree with what the screen shows. press @ref reports success and taps the wrong place.

The runner must publish one coordinate space for a window that does not fill the screen. A tap on a ref must land on that ref's control.

This bug predates #3895. #3895 refuses host AX bridge captures in this state (app-window-not-full-screen) and sends them to XCTest. After #3895, XCTest is the only path that serves a windowed iPad app, so this bug has no fallback.

Observed

iPad Pro 11-inch (M5) simulator, iOS 27.0, Xcode 27.1, screen 834×1210 pt. Source origin/main 9935ef51c2 (agent-device 0.21.31-dev, no #3895). Windowed Apps was the Settings default. The window was shrunk with gesture pan 822 1198 -230 -330 from the resize handle while the app was full-screen. Every capture below is backend: tree (XCTest).

Full screen (control)

Calendar full screen: viewport 834×1210. The add button rect is (192, 36, 36×36). The screenshot shows + at the same place. press @ref hit the right control every time (Settings rows, alert buttons).

Windowed

Calendar window: on-screen box (224, 207, 386×682), measured in the screenshot.

1. Raw frames are window-relative. snapshot --raw reports the application at (0, 0, 386×682). The Add button is (335, 14, 37×36), and its screen position is (559, 221). Every raw rect needs the window origin (224, 207) added to match the screen.

2. The viewport is the window clipped against the wrong space. Regular snapshot -i reports viewport 162×475. That is 386−224 by 682−207: the window-relative frame cut by the screen-space window origin. The Settings window (375×486 at (230, 305)) gives 145×181 the same way. The #3895 worker saw 152×196.

3. Clipping drops controls that are on screen. Calendar: clippedNodeCount 27, droppedNodeCount 43. The toolbar (October, Single Day, Search, Add), the day strip, and Today are missing from the tree. They are all visible in the screenshot. Settings: 20 clipped, 14 dropped, and the back button is gone.

4. Surviving rects are wrong by the window origin. Inbox is reported at (327.5, 632, 44.5×36) with hittable: true. It is on screen at (551.5, 839).

5. The capture can fail. With the Calendar grid scrolled to 3 AM, snapshot, snapshot -i, and find all failed on every retry:

   COMMAND_FAILED  regular iOS snapshot node escaped its cumulative clip
   details.reason: regular-node-outside-cumulative-clip
   index 2, frame (224, 207, 386×682), clip (224, 207, 162×475)
   

   After a relaunch moved the grid to 5 AM, the same window gave a clipped tree instead (observation 2). The outcome depends on the tree.

6. Coordinate taps land at offset − window origin. The runner anchors at the app origin (interactionCoordinate). Probes on the Calendar window (origin 224, 207):

   press x y	Meaning	Result
   577 239	screen position of +	nothing
   354 33	window-relative centre of +	nothing
   193 102	screen position of Wed 21 minus origin	nothing
   641 516	Wed 21 screen position plus origin	Wed 21 selected
   530 516	Mon 19 screen position plus origin	Mon 19 selected
   801 446	+ screen position plus origin	New Event sheet opened

   The tap lands at (x − 224, y − 207). A tap on screen point S needs offset S + origin.

7. press @ref reports success and misses.

   • Settings: press @e8 (Stage Manager row) tapped (287, 415) and returned success. The Stage Manager row is at about (418, 738) on screen, and by the model in item 6 the tap landed near (57, 110) on the wallpaper. Nothing changed.
   • Calendar: press @e21 (Inbox) tapped (350, 650) and returned success. By the same model the tap landed near (126, 443) on the wallpaper. Nothing opened.
   • The tapped point is the centre of the clipped, window-relative rect. The point is wrong by one origin in the rect and by one more origin in the dispatch.

8. Other commands use the same wrong geometry:

   • gesture pan and swipe fail with INVALID_ARGS "Gesture trajectory does not fit inside the viewport", because the viewport is 145×181.
   • press x y warns "outside the last-known viewport (386x682)" for a point that is inside the window on screen.
   • fill @e3 on a focused search field fails with NO_TEXT_INPUT_AT_POINT.
   • type typed one character and then returned TEXT_INPUT_NOT_FOCUSED, although the field was focused. The text was typed one character per call.
   • scroll down in the Calendar window lost the runner transport. Afterwards the simulator showed the Home Screen page and the Calendar window was gone. I did not isolate the cause.

Not reproduced here: Stage Manager (the #3895 worker reports the same offset there) and Slide Over (untested).

Expected

For a window that does not fill the screen:

• Every published rect and the viewport are in one coordinate space. This is the principle of the ADR 0004 amendment "The coordinate space of a captured subtree (#2612)": one capture publishes one space.
• That space is screen space in points. A rect matches the position of its control in screenshot, as it does in full screen.
• The viewport is the window's visible box, for example (224, 207, 386×682).
• A control inside the window is never dropped for being "outside the clip". The capture does not fail with regular-node-outside-cumulative-clip because of the window origin.
• press @ref and press x y land on the same control the rect names.
• Gesture, scroll, fill, and type bounds use the same space and the same viewport.
• Full-screen captures, and the foldable panel taps of ADR 0025, are unchanged.

Required behavior

• Pick one space for a window that does not fill the screen. Name it in ADR 0004. Convert the runner's window-relative element frames into it where the runner builds the raw tree.
• Build the viewport from the same space as the rects. Remove the mixed-space clip that produces (w − x) × (h − y).
• Dispatch coordinate taps, long presses, drags, and gestures so that the point a caller passes is the point that is touched. Account for the window origin in interactionCoordinate. Keep the foldable display selection that resolveRunnerWindow provides.
• Keep behavior unchanged when the window frame equals the screen frame.
• Do not lift #3895's app-window-not-full-screen refusal here. The host AX bridge still reports window-relative frames. Lifting the refusal is a separate decision.
• Use typed reasons. Do not match error text.
• Update ADR 0004, the snapshot docs, and the press help text if the described behavior changes.

Completion conditions

All checks run on an iPad Simulator with Windowed Apps on and a shrunken Calendar window, and on the same window after a move.

1. snapshot -i --json returns success in every scroll state of the Calendar grid. No regular-node-outside-cumulative-clip.
2. viewport equals the window's on-screen box within 1 pt. The toolbar buttons, the day strip, and Today are in the tree. No node the screenshot shows is dropped.
3. For at least five nodes, each rect matches its on-screen position within 1 pt. Add is within 1 pt of (559, 221) for a window at (224, 207).
4. press @<Add> opens the New Event sheet. press @<Wed 21> selects Wed 21. press @<Inbox> opens the inbox. Use --settle or a screenshot as proof.
5. press <x> <y> with the screen position of Add opens the New Event sheet.
6. fill and type into a focused field in the window succeed. gesture pan inside the window is accepted.
7. A Full Screen Apps run produces rects and tap results identical to before the change. The foldable tests for ADR 0025 stay green.
8. A new unit test covers the coordinate conversion and the viewport for a window that does not fill the screen. It fails on main before the fix. The window-coordinate-space fixture contracts/fixtures/window-coordinate-space.json gains a windowed case if the Swift and TypeScript parity replay applies.
9. Device evidence for the PR lists the commands, simulator, source SHA, and the results of checks 1 to 6. Follow docs/agents/device-verification.md.

Repro

pnpm install --frozen-lockfile && pnpm build
UDID=$(xcrun simctl create "Windowed Geometry Worker" com.apple.CoreSimulator.SimDeviceType.iPad-Pro-11-inch-M5-12GB com.apple.CoreSimulator.SimRuntime.iOS-27-0)
xcrun simctl boot $UDID
export AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH=$PWD/.tmp/ios-runner-derived
S="--state-dir /private/tmp/windowed-geometry-worker --udid $UDID --platform ios --session wgw"
node bin/agent-device.mjs open com.apple.mobilecal $S --foreground     # first run builds the runner (about 1 min)
# dismiss the welcome and permission alerts with press @ref; the app is full screen
node bin/agent-device.mjs gesture pan 822 1198 -230 -330 800 $S         # shrink from the resize handle
node bin/agent-device.mjs screenshot window.png $S                      # window at about (224, 207, 386x682)
node bin/agent-device.mjs snapshot -i --json $S                         # viewport 162x475, or the cumulative-clip failure
node bin/agent-device.mjs snapshot --raw --json $S                      # window-relative frames
node bin/agent-device.mjs press 801 446 $S                              # opens New Event: tap lands at offset − origin
node bin/agent-device.mjs close $S && node bin/agent-device.mjs daemon stop --state-dir /private/tmp/windowed-geometry-worker
xcrun simctl delete $UDID

The Settings app (com.apple.Preferences) shows the same drift. Any app works: examples/test-app is not needed.

Dependencies

• Related to #3895 (open). It adds the app-window-not-full-screen refusal, which routes these captures to this runner. Its verification table records "runner offset" for Windowed and Stage Manager. This issue is not blocked by #3895. Rebase on it or wait for it to merge so the ADR 0004 text does not conflict.
• Related to #3844 (iPad snapshot routing and XCTest channel behavior). #3895 refs it. This issue does not change channel selection.
• Related to #3801 (closed): an earlier case of the same regular-node-outside-cumulative-clip failure from different geometry.
• Related to ADR 0025 and #2711: foldable panel taps use the window and display resolution in resolveRunnerWindow. The fix must keep them.

Code owners and paths

• apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotAcquisition.swift: snapshotAppFrame and safeSnapshotViewport publish the screen-space window frame (onScreenWindowFrame) as the viewport box. The element frames in the same tree are window-relative.
• apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift: resolveRunnerWindow (line 261), onScreenWindowFrame (line 281), and interactionCoordinate (line 618). The iOS branch anchors at app.coordinate(withNormalizedOffset: 0, 0). Its comment says the snapshot hands back app-space points.
• apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift: the reference frame (referenceWidth) that the daemon uses for the press response.
• apple/snapshot-presentation/Sources/AgentDeviceSnapshotPresentation/ (SnapshotGeometry.swift, SnapshotCoordinateSpace.swift, SnapshotPresentationInvariant.swift) and packages/capture-kit/src/ios-snapshot-engine/ (geometry.ts, geometry-policy.ts, invariants.ts): the clip and the regular-node-outside-cumulative-clip invariant.
• packages/contracts/src/gesture-plan.ts (line 496): the gesture bound check. src/commands/interaction/runtime/resolution.ts (line 76): the viewport warning. Both read the published viewport and need no change if the viewport is correct.
• docs/adr/0004-ios-snapshot-backend-strategy.md: the coordinate-space amendment (#2612) to extend. Run pnpm check:xctest-selection and build the runner (pnpm build:xcuitest) for the Swift change.