fix(ios): use the resolved display for screenshots and visual verification
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 38/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- ios, typescript
Línea de trabajo
Start with AGENTS.md, PR #2724's window-resolution order, and packages/platform-apple/src/core/screenshot.ts; then trace the iOS screenshot, captureRunnerFrame, and navigation fallback paths. Validate the capture contract and encoding with focused tests, native builds/XCTest selection checks, affected gates, and the listed pose matrix. Done means metadata follows the runner result, invalid windows follow the stated failure contracts, and macOS/tvOS behavior remains unchanged.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Part of #2725. Investigation evidence and reproducible probe: https://github.com/callstack/agent-device/issues/2727#issuecomment-5762778410.
Resolved design / dependencies
Use the resolved app window's screen, not XCUIScreen.main, for iOS app-screen capture and visual verification. Resolve app.windows.firstMatch.frame before obtaining that window's private screen; screenshot the resulting XCUIScreen. Open, half-open, closed, a single-panel control, and a native fixture alert were measured successfully. Do not choose by a hardcoded display number or the first screen in XCUIScreen.screens.
Prerequisite: PR #2724; implement on its branch with an explicit PR base, or after it merges. No unresolved design decision remains. Consumer-level checks below remain required implementation validation; the spike proves the capture primitive, not the complete migration.
Small owning interface
Introduce one iOS capture helper, conceptually captureResolvedAppScreen(app:) -> CapturedAppScreen, returning an upright image and actual display ID, encoded pixel width/height, and pixels-per-point. Keep the private window/screen lookup exception-safe in the native XCTest bridge. Reuse the window-resolution order established by #2724; do not maintain independently drifting display resolvers or introduce a session-wide display cache.
Normalize the image by drawing it in its oriented logical size with its actual scale before PNG encoding. Measured open Duo: window 951x669, UIImage logical size about 950.667x668.667, scale 3, raw CGImage/PNG 2006x2852, UIImage orientation right; upright drawing yields 2852x2006. Do not infer orientation from raw CGImage dimensions, rely on PNG metadata being respected by every consumer, or stretch the one-pixel XCTest discrepancy to CoreDevice's nominal 2853x2007 size.
Wire and failure contracts
Carry screenshot-only metadata with the runner screenshot result (e.g. screenshotMetadata: {displayID, pixelWidth, pixelHeight, pixelsPerPoint}), alongside the existing file/base64 payload. Declare it at the owning cross-language contract, not by per-consumer casts. Host normalization uses those actual source facts, never SIMULATOR_MAINSCREEN_SCALE for an image from another panel.
An unresolved/invalid app window or screen fails required screenshot capture with a typed reason, without selecting main. Optional visual checks return their existing unavailable/unknown result; no sample is not evidence of stability or navigation change. Do not make an observation activate an app merely to obtain a screen; use the caller's intended app context.
Bounded migration
- Route the iOS screenshot case,
captureRunnerFrame, and navigation fallback visual capture through this helper. Keyboard settling already consumes frame capture and keeps its separate keyboard visibility check. - Update
packages/platform-apple/src/core/screenshot.tsrunner fallback normalization to consume source metadata. - Preserve macOS/tvOS behavior. Do not redesign gestures, text entry, host display inventory, or recording touch overlays (#2707).
- Follow the repository split-before-behavior rule for oversized sources, with mechanical structural changes separated from behavior.
Acceptance and validation
- Force simctl screenshot failure: runner fallback captures the visible inner content upright at correct density.
- Navigation change/no-change controls and keyboard settling observe the lit display. A failed sample cannot satisfy a positive visual assertion.
- Test open, half-open, closed, and a single-panel control; preserve a visible native overlay in the full-screen capture. Record pose, display, image orientation, actual pixel dimensions, scale, and app state.
- Repeat the pose matrix on the implementation PR head. During the spike, moving Device Hub from the second host display to the main display cleared
device-hub-window-missing; document this troubleshooting detail rather than treating it as a missing simulator. - Add focused contract/encoding tests, native builds and XCTest selection checks, affected gates, and local semantic evidence. Update ADR 0025 only to claims proved by those checks.
Ready for implementation with the explicit #2724 branch/merge prerequisite. Follow AGENTS.md; keep one task per PR and clean up manual sessions.
- Lenguaje dominante
- TypeScript
- Estrellas
- 4.7k
- Forks
- 304
- Merge medio
- 12 h 42 min
- PR fusionados (30 d)
- 463
Guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de callstack/agent-device
-
mutation-model: indirect-ownership self-test pins a test that stopped reaching scroll-edge-state Abiertobug needs-triage
Dificultad 2/5 1-3 horas Aptitud para principiantes 88/100
callstack/agent-device#2547 ·
-
refactor
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
callstack/agent-device#2530 · 1 comentario ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
callstack/agent-device#1869 ·
-
enhancement ready-for-agent
Dificultad 4/5 3-5 días Aptitud para principiantes 42/100
callstack/agent-device#2731 ·
-
bug ready-for-agent
Dificultad 3/5 1-2 días Aptitud para principiantes 72/100
callstack/agent-device#2729 ·
Todos los issues de callstack/agent-device
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 84/100
Eynzof/Hermes-CN-Desktop#610 ·
-
[Bug]: Matrix progress drafts fail with "Matrix runtime not initialized" during tool activity Abiertobug clawsweeper:linked-pr-open clawsweeper:needs-live-repro clawsweeper:no-new-fix-pr impact:message-loss issue-rating: 🐚 platinum hermit P2 regression
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
-
Client support matrix inclusion Abiertoenhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
-
calcite-components needs triage refactor
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
Esri/calcite-design-system#15203 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 90/100
danielmiessler/LifeOS#2218 ·