iOS: fill/type with coordinates crashes the runner when the app AX root snapshot is rejected (XCPointerEventPath assertion)

Offen
  • Stimmt etwas nicht?
#4,274 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
48/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
ios, swift, typescript
Bereich
mobile, testing-qa

Rechercherichtung

Start with RunnerTests+TextEntryFocus.swift and trace focusTextInputForTextEntry through the synthesized gesture and text-entry routes. Check how synthesizedCoordinateContext reports unavailable accessibility health and how the replacement route is selected; the issue also flags the sole-screen reference frame. Done means coordinate text entry does not crash the runner when AX snapshots are rejected, and the reported frame behavior is investigated.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Written by an AI agent on behalf of @gursheyss.

Problem

On iOS, fill / type with coordinates kills the XCUITest runner when the app's accessibility root can't be snapshotted. The daemon then reports:

ENGINE_FAILURE perform fill failed: Runner command "type" lost its transport response and the status probe failed

tap on the same screen keeps working, because it routes through the synthesized coordinate tap. focusTextInputForTextEntry (RunnerTests+TextEntryFocus.swift) calls tapAt(app:x:y:) instead, and that uses XCUICoordinate. With the app root rejected, XCTest asserts inside the coordinate tap and the runner's testCommand dies.

Environment

  • agent-device 0.21.36 (installed from npm, driven by e2e 0.21.3)
  • iPhone 17 Pro simulator, iOS 26.5, macOS arm64 host
  • React Native (Expo) app

Trigger

On the failing screen, the app's accessibility tree is 62 levels deep, which is XCTest's snapshot depth limit. A raw agent-device snapshot --raw --json still succeeds there and reports max depth 62 with 7 nodes at that depth. Every XCUIApplication query, though, starts recording:

Failed to resolve query: Error kAXErrorIllegalArgument getting snapshot for element <AXUIElementRef …> {pid=<app>} {uid=[ID:1 hash:0x0]}

It begins right after a reply that renders a horizontally scrolling table inside a chat bubble, which adds about 6 levels. Screens below the limit don't show it.

Runner log (before)

AGENT_DEVICE_RUNNER_COMMAND_ACCEPTED command=tap …
AGENT_DEVICE_RUNNER_AX_SNAPSHOT_ISSUE_SUPPRESSED … kAXErrorIllegalArgument getting snapshot for element … {uid=[ID:1 hash:0x0]}
AGENT_DEVICE_RUNNER_SYNTHESIZED_SOLE_SCREEN_ROUTE resolved={{0, 0}, {320, 480}}
AGENT_DEVICE_RUNNER_SYNTHESIZED_GESTURE_POLICY kind=coordinateTap axHealth=unavailable fallbackAllowed=true fallbackAttempted=false
AGENT_DEVICE_RUNNER_COMMAND_COMPLETED command=tap … ok=1
AGENT_DEVICE_RUNNER_COMMAND_ACCEPTED command=type …
AGENT_DEVICE_RUNNER_RECORDED_ISSUE_HELD issue=Failed to resolve query: Error kAXErrorIllegalArgument getting snapshot for element …
No element snapshot found for Application … -> (201.0, 462.0)
*** Assertion failure in -[XCPointerEventPath initForTouchAtPoint:offset:], XCSynthesizedEventRecord.m:282

After this the runner never answers the type command, and the daemon reboots it, losing the rest of the session.

Workaround we're carrying (pnpm patch)

In focusTextInputForTextEntry, when synthesizedCoordinateContext(app:) reports accessibilityHealth == .unavailable, focus with performSynthesizedGesture(.coordinateTap) / synthesizedTapAt. Only if that isn't .performed does it fall back to tapAt:

var focusTapped = false
#if os(iOS)
let context = synthesizedCoordinateContext(app: app)
if context?.accessibilityHealth == .unavailable,
  case .performed = performSynthesizedGesture(app, kind: .coordinateTap, context: context, synthesize: {
    synthesizedTapAt(app: app, x: tapPoint.x, y: tapPoint.y, context: context)
  })
{
  focusTapped = true
}
#endif
if !focusTapped {
  _ = tapAt(app: app, x: tapPoint.x, y: tapPoint.y)
}

With it, 7 of 7 runs kept the runner alive, with 0 transport losses (before: 4 of 7 died this way). fill still fails, but now cleanly and with a clear message:

perform fill failed: no text input found at the provided coordinates to clear

That's because textInputAt, the keyboard check and read-back all go through XCTest queries, which fail the same way. The replacement route that skips them (shouldUseResolvedCoordinateTextEntryRoute / shouldUseSynthesizedFirstResponderReplacement) only turns on when the XCTest channel is penalized, not when accessibilityHealth is .unavailable.

Asks

  1. Don't call the XCUICoordinate tap from text-entry focus when AX health is unavailable. A runner crash takes down every later command in the session.
  2. Consider letting the synthesized first-responder replacement route also cover accessibilityHealth == .unavailable, so fill can work on screens past the depth limit.
  3. Separately, the sole-screen reference frame came back as {{0, 0}, {320, 480}} on a 402×874 device. That looks wrong, though taps still landed.

We haven't built a standalone repro app. The cianru/deep-view-snapshot-poc example (136 levels) is likely enough to show the same rejection.

Vorherrschende Sprache
TypeScript
Sterne
5.3k
Forks
377
Ø Merge
13 Std. 58 Min.
Gemergte PRs (30 T.)
654

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus callstack/agent-device

Alle Issues in callstack/agent-device

Ähnliche Issues

Weitere Issues zu TypeScript

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.