iOS: fill/type with coordinates crashes the runner when the app AX root snapshot is rejected (XCPointerEventPath assertion)

Đang mở
  • Có gì không ổn?
#4,274 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
48/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
ios, swift, typescript
Lĩnh vực
mobile, testing-qa

Hướng nghiên cứu

Start with RunnerTests+TextEntryFocus.swift and trace focusTextInputForTextEntry through the synthesized gesture and text-entry routes. Check how synthesizedCoordinateContext reports unavailable accessibility health and how the replacement route is selected; the issue also flags the sole-screen reference frame. Done means coordinate text entry does not crash the runner when AX snapshots are rejected, and the reported frame behavior is investigated.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Written by an AI agent on behalf of @gursheyss.

Problem

On iOS, fill / type with coordinates kills the XCUITest runner when the app's accessibility root can't be snapshotted. The daemon then reports:

ENGINE_FAILURE perform fill failed: Runner command "type" lost its transport response and the status probe failed

tap on the same screen keeps working, because it routes through the synthesized coordinate tap. focusTextInputForTextEntry (RunnerTests+TextEntryFocus.swift) calls tapAt(app:x:y:) instead, and that uses XCUICoordinate. With the app root rejected, XCTest asserts inside the coordinate tap and the runner's testCommand dies.

Environment

  • agent-device 0.21.36 (installed from npm, driven by e2e 0.21.3)
  • iPhone 17 Pro simulator, iOS 26.5, macOS arm64 host
  • React Native (Expo) app

Trigger

On the failing screen, the app's accessibility tree is 62 levels deep, which is XCTest's snapshot depth limit. A raw agent-device snapshot --raw --json still succeeds there and reports max depth 62 with 7 nodes at that depth. Every XCUIApplication query, though, starts recording:

Failed to resolve query: Error kAXErrorIllegalArgument getting snapshot for element <AXUIElementRef …> {pid=<app>} {uid=[ID:1 hash:0x0]}

It begins right after a reply that renders a horizontally scrolling table inside a chat bubble, which adds about 6 levels. Screens below the limit don't show it.

Runner log (before)

AGENT_DEVICE_RUNNER_COMMAND_ACCEPTED command=tap …
AGENT_DEVICE_RUNNER_AX_SNAPSHOT_ISSUE_SUPPRESSED … kAXErrorIllegalArgument getting snapshot for element … {uid=[ID:1 hash:0x0]}
AGENT_DEVICE_RUNNER_SYNTHESIZED_SOLE_SCREEN_ROUTE resolved={{0, 0}, {320, 480}}
AGENT_DEVICE_RUNNER_SYNTHESIZED_GESTURE_POLICY kind=coordinateTap axHealth=unavailable fallbackAllowed=true fallbackAttempted=false
AGENT_DEVICE_RUNNER_COMMAND_COMPLETED command=tap … ok=1
AGENT_DEVICE_RUNNER_COMMAND_ACCEPTED command=type …
AGENT_DEVICE_RUNNER_RECORDED_ISSUE_HELD issue=Failed to resolve query: Error kAXErrorIllegalArgument getting snapshot for element …
No element snapshot found for Application … -> (201.0, 462.0)
*** Assertion failure in -[XCPointerEventPath initForTouchAtPoint:offset:], XCSynthesizedEventRecord.m:282

After this the runner never answers the type command, and the daemon reboots it, losing the rest of the session.

Workaround we're carrying (pnpm patch)

In focusTextInputForTextEntry, when synthesizedCoordinateContext(app:) reports accessibilityHealth == .unavailable, focus with performSynthesizedGesture(.coordinateTap) / synthesizedTapAt. Only if that isn't .performed does it fall back to tapAt:

var focusTapped = false
#if os(iOS)
let context = synthesizedCoordinateContext(app: app)
if context?.accessibilityHealth == .unavailable,
  case .performed = performSynthesizedGesture(app, kind: .coordinateTap, context: context, synthesize: {
    synthesizedTapAt(app: app, x: tapPoint.x, y: tapPoint.y, context: context)
  })
{
  focusTapped = true
}
#endif
if !focusTapped {
  _ = tapAt(app: app, x: tapPoint.x, y: tapPoint.y)
}

With it, 7 of 7 runs kept the runner alive, with 0 transport losses (before: 4 of 7 died this way). fill still fails, but now cleanly and with a clear message:

perform fill failed: no text input found at the provided coordinates to clear

That's because textInputAt, the keyboard check and read-back all go through XCTest queries, which fail the same way. The replacement route that skips them (shouldUseResolvedCoordinateTextEntryRoute / shouldUseSynthesizedFirstResponderReplacement) only turns on when the XCTest channel is penalized, not when accessibilityHealth is .unavailable.

Asks

  1. Don't call the XCUICoordinate tap from text-entry focus when AX health is unavailable. A runner crash takes down every later command in the session.
  2. Consider letting the synthesized first-responder replacement route also cover accessibilityHealth == .unavailable, so fill can work on screens past the depth limit.
  3. Separately, the sole-screen reference frame came back as {{0, 0}, {320, 480}} on a 402×874 device. That looks wrong, though taps still landed.

We haven't built a standalone repro app. The cianru/deep-view-snapshot-poc example (136 levels) is likely enough to show the same rejection.

Ngôn ngữ chính
TypeScript
Star
5.3k
Fork
377
Merge trung bình
14 giờ 6 phút
Pull request đã merge (30 ngày)
650

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của callstack/agent-device

Tất cả issue của callstack/agent-device

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.