fix(ios): use the resolved display for screenshots and visual verification

Abierto
#2,728 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
38/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
ios, typescript
Área
mobile, testing

Línea de trabajo

Start with AGENTS.md, PR #2724's window-resolution order, and packages/platform-apple/src/core/screenshot.ts; then trace the iOS screenshot, captureRunnerFrame, and navigation fallback paths. Validate the capture contract and encoding with focused tests, native builds/XCTest selection checks, affected gates, and the listed pose matrix. Done means metadata follows the runner result, invalid windows follow the stated failure contracts, and macOS/tvOS behavior remains unchanged.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug ready-for-agent

Part of #2725. Investigation evidence and reproducible probe: https://github.com/callstack/agent-device/issues/2727#issuecomment-5762778410.

Resolved design / dependencies

Use the resolved app window's screen, not XCUIScreen.main, for iOS app-screen capture and visual verification. Resolve app.windows.firstMatch.frame before obtaining that window's private screen; screenshot the resulting XCUIScreen. Open, half-open, closed, a single-panel control, and a native fixture alert were measured successfully. Do not choose by a hardcoded display number or the first screen in XCUIScreen.screens.

Prerequisite: PR #2724; implement on its branch with an explicit PR base, or after it merges. No unresolved design decision remains. Consumer-level checks below remain required implementation validation; the spike proves the capture primitive, not the complete migration.

Small owning interface

Introduce one iOS capture helper, conceptually captureResolvedAppScreen(app:) -> CapturedAppScreen, returning an upright image and actual display ID, encoded pixel width/height, and pixels-per-point. Keep the private window/screen lookup exception-safe in the native XCTest bridge. Reuse the window-resolution order established by #2724; do not maintain independently drifting display resolvers or introduce a session-wide display cache.

Normalize the image by drawing it in its oriented logical size with its actual scale before PNG encoding. Measured open Duo: window 951x669, UIImage logical size about 950.667x668.667, scale 3, raw CGImage/PNG 2006x2852, UIImage orientation right; upright drawing yields 2852x2006. Do not infer orientation from raw CGImage dimensions, rely on PNG metadata being respected by every consumer, or stretch the one-pixel XCTest discrepancy to CoreDevice's nominal 2853x2007 size.

Wire and failure contracts

Carry screenshot-only metadata with the runner screenshot result (e.g. screenshotMetadata: {displayID, pixelWidth, pixelHeight, pixelsPerPoint}), alongside the existing file/base64 payload. Declare it at the owning cross-language contract, not by per-consumer casts. Host normalization uses those actual source facts, never SIMULATOR_MAINSCREEN_SCALE for an image from another panel.

An unresolved/invalid app window or screen fails required screenshot capture with a typed reason, without selecting main. Optional visual checks return their existing unavailable/unknown result; no sample is not evidence of stability or navigation change. Do not make an observation activate an app merely to obtain a screen; use the caller's intended app context.

Bounded migration

  • Route the iOS screenshot case, captureRunnerFrame, and navigation fallback visual capture through this helper. Keyboard settling already consumes frame capture and keeps its separate keyboard visibility check.
  • Update packages/platform-apple/src/core/screenshot.ts runner fallback normalization to consume source metadata.
  • Preserve macOS/tvOS behavior. Do not redesign gestures, text entry, host display inventory, or recording touch overlays (#2707).
  • Follow the repository split-before-behavior rule for oversized sources, with mechanical structural changes separated from behavior.

Acceptance and validation

  • Force simctl screenshot failure: runner fallback captures the visible inner content upright at correct density.
  • Navigation change/no-change controls and keyboard settling observe the lit display. A failed sample cannot satisfy a positive visual assertion.
  • Test open, half-open, closed, and a single-panel control; preserve a visible native overlay in the full-screen capture. Record pose, display, image orientation, actual pixel dimensions, scale, and app state.
  • Repeat the pose matrix on the implementation PR head. During the spike, moving Device Hub from the second host display to the main display cleared device-hub-window-missing; document this troubleshooting detail rather than treating it as a missing simulator.
  • Add focused contract/encoding tests, native builds and XCTest selection checks, affected gates, and local semantic evidence. Update ADR 0025 only to claims proved by those checks.

Ready for implementation with the explicit #2724 branch/merge prerequisite. Follow AGENTS.md; keep one task per PR and clean up manual sessions.

Lenguaje dominante
TypeScript
Estrellas
4.7k
Forks
304
Merge medio
12 h 42 min
PR fusionados (30 d)
463

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de callstack/agent-device

Todos los issues de callstack/agent-device

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.