iOS: fill/type with coordinates crashes the runner when the app AX root snapshot is rejected (XCPointerEventPath assertion)

Ouverte
  • Quelque chose ne va pas ?
#4,274 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
48/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
ios, swift, typescript
Domaine
mobile, testing-qa

Piste de recherche

Start with RunnerTests+TextEntryFocus.swift and trace focusTextInputForTextEntry through the synthesized gesture and text-entry routes. Check how synthesizedCoordinateContext reports unavailable accessibility health and how the replacement route is selected; the issue also flags the sole-screen reference frame. Done means coordinate text entry does not crash the runner when AX snapshots are rejected, and the reported frame behavior is investigated.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Written by an AI agent on behalf of @gursheyss.

Problem

On iOS, fill / type with coordinates kills the XCUITest runner when the app's accessibility root can't be snapshotted. The daemon then reports:

ENGINE_FAILURE perform fill failed: Runner command "type" lost its transport response and the status probe failed

tap on the same screen keeps working, because it routes through the synthesized coordinate tap. focusTextInputForTextEntry (RunnerTests+TextEntryFocus.swift) calls tapAt(app:x:y:) instead, and that uses XCUICoordinate. With the app root rejected, XCTest asserts inside the coordinate tap and the runner's testCommand dies.

Environment

  • agent-device 0.21.36 (installed from npm, driven by e2e 0.21.3)
  • iPhone 17 Pro simulator, iOS 26.5, macOS arm64 host
  • React Native (Expo) app

Trigger

On the failing screen, the app's accessibility tree is 62 levels deep, which is XCTest's snapshot depth limit. A raw agent-device snapshot --raw --json still succeeds there and reports max depth 62 with 7 nodes at that depth. Every XCUIApplication query, though, starts recording:

Failed to resolve query: Error kAXErrorIllegalArgument getting snapshot for element <AXUIElementRef …> {pid=<app>} {uid=[ID:1 hash:0x0]}

It begins right after a reply that renders a horizontally scrolling table inside a chat bubble, which adds about 6 levels. Screens below the limit don't show it.

Runner log (before)

AGENT_DEVICE_RUNNER_COMMAND_ACCEPTED command=tap …
AGENT_DEVICE_RUNNER_AX_SNAPSHOT_ISSUE_SUPPRESSED … kAXErrorIllegalArgument getting snapshot for element … {uid=[ID:1 hash:0x0]}
AGENT_DEVICE_RUNNER_SYNTHESIZED_SOLE_SCREEN_ROUTE resolved={{0, 0}, {320, 480}}
AGENT_DEVICE_RUNNER_SYNTHESIZED_GESTURE_POLICY kind=coordinateTap axHealth=unavailable fallbackAllowed=true fallbackAttempted=false
AGENT_DEVICE_RUNNER_COMMAND_COMPLETED command=tap … ok=1
AGENT_DEVICE_RUNNER_COMMAND_ACCEPTED command=type …
AGENT_DEVICE_RUNNER_RECORDED_ISSUE_HELD issue=Failed to resolve query: Error kAXErrorIllegalArgument getting snapshot for element …
No element snapshot found for Application … -> (201.0, 462.0)
*** Assertion failure in -[XCPointerEventPath initForTouchAtPoint:offset:], XCSynthesizedEventRecord.m:282

After this the runner never answers the type command, and the daemon reboots it, losing the rest of the session.

Workaround we're carrying (pnpm patch)

In focusTextInputForTextEntry, when synthesizedCoordinateContext(app:) reports accessibilityHealth == .unavailable, focus with performSynthesizedGesture(.coordinateTap) / synthesizedTapAt. Only if that isn't .performed does it fall back to tapAt:

var focusTapped = false
#if os(iOS)
let context = synthesizedCoordinateContext(app: app)
if context?.accessibilityHealth == .unavailable,
  case .performed = performSynthesizedGesture(app, kind: .coordinateTap, context: context, synthesize: {
    synthesizedTapAt(app: app, x: tapPoint.x, y: tapPoint.y, context: context)
  })
{
  focusTapped = true
}
#endif
if !focusTapped {
  _ = tapAt(app: app, x: tapPoint.x, y: tapPoint.y)
}

With it, 7 of 7 runs kept the runner alive, with 0 transport losses (before: 4 of 7 died this way). fill still fails, but now cleanly and with a clear message:

perform fill failed: no text input found at the provided coordinates to clear

That's because textInputAt, the keyboard check and read-back all go through XCTest queries, which fail the same way. The replacement route that skips them (shouldUseResolvedCoordinateTextEntryRoute / shouldUseSynthesizedFirstResponderReplacement) only turns on when the XCTest channel is penalized, not when accessibilityHealth is .unavailable.

Asks

  1. Don't call the XCUICoordinate tap from text-entry focus when AX health is unavailable. A runner crash takes down every later command in the session.
  2. Consider letting the synthesized first-responder replacement route also cover accessibilityHealth == .unavailable, so fill can work on screens past the depth limit.
  3. Separately, the sole-screen reference frame came back as {{0, 0}, {320, 480}} on a 402×874 device. That looks wrong, though taps still landed.

We haven't built a standalone repro app. The cianru/deep-view-snapshot-poc example (136 levels) is likely enough to show the same rejection.

Langage dominant
TypeScript
Étoiles
5.3k
Forks
377
Merge moyen
14 h 6 min
PR mergées (30 j)
650

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de callstack/agent-device

Toutes les issues de callstack/agent-device

Issues similaires

Plus d'issues TypeScript

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.