Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

iOS: XCTest-channel penalty keeps the runner in private-AX fallback most of a long session (fixed 3 s / 120 s, no override)

Open
  • Is something wrong?
#3,844 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
ios, typescript
Domain
mobile, performance, testing

Research direction

Read the snapshot penalty and capture plan code under dist/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/, starting with RunnerTests+SnapshotTiming.swift, RunnerTests+SnapshotCapturePlan.swift, and the snapshot CLI entry point. Check the related issue #1542 and presentedSurface work in #3565 before choosing a scope. Done means the requested penalty controls and CLI backend option behave as specified, with tests covering single query timeouts and recovery probes.

Written by the indexing model from the issue text.

Description

bug

Summary

On a heavy UIKit app (iPad simulator), one slow or timed-out XCTest-channel snapshot tier puts the runner into private-AX fallback for 120 s. A timed-out 1 s recursiveTree recovery probe re-arms it; that probe runs when the private-AX capture is rejected. And once the penalty expires, the next capture's 1 s query sweep often times out and penalizes the channel again. In one long-running session log, 71 % of snapshot commands ran with the XCTest channel deferred.

In our captures, private-AX returned presented screens together with the screen underneath them. A test then can't reliably tell which screen is top-most, and an element of the covered screen looks "visible". For read-only checks this is mostly harmless. For steps that must act on the right screen (bulk select / destructive bulk actions in a list), it's unsafe, so we refuse those steps whenever a fallback happened in the last 120 s.

Version / environment

  • agent-device 0.21.25 (pinned by @e2e-dev/mobile 0.12.0, used through the e2e 0.20.0 runner). The penalty logic is unchanged in 0.21.28 and 0.21.29-nightly.20261020.
  • Xcode 27.0 (27A266a), iOS 27.0 simulator runtime, iPad Air 11-inch (M4) simulators
  • UIKit app with long table views

What the source shows (0.21.25, dist/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/)

  • An XCTest-channel tier penalizes the channel when it times out, exhausts its deadline, or takes longer than snapshotXCTestSlowCaptureThreshold (3 s): RunnerTests+SnapshotTiming.swift:119-155, constant at RunnerTests.swift:139. Budgets: tree slice 8 s (RunnerTests.swift:143), query sweep 1.0 s (RunnerTests+Snapshot.swift:134-143), viewport read ≤ 1.0 s (RunnerTests+SnapshotAcquisition.swift:50-56). A system-modal probe timeout also penalizes (RunnerTests+Snapshot.swift:360).
  • penalizeSnapshotXCTestChannel sets penaltyUntil = now + 120 s (RunnerTests+SnapshotCapturePlan.swift:133-143, RunnerTests.swift:122). While it's active, the plan puts private-AX first, plus recursiveTree with a 1 s slice (RunnerTests+SnapshotCapturePlan.swift:110, 256-277).
  • Each slow or timed-out XCTest attempt during the penalty re-arms it. In practice that's the 1 s tree probe, which runs when the private-AX tier fails or is rejected (RunnerTests+SnapshotCapturePlan.swift:272, 391-431). penaltySuppressed applies only to the one-shot warm-up after a target PID change (:304, 373-378; RunnerTests+Lifecycle.swift:254).
  • It's cleared only by targetReset / a new target PID (RunnerTests+Lifecycle.swift:228, 252), by the 120 s expiring, or by a runner restart.
  • The thresholds and the duration are let constants. No env var, config or CLI flag changes them. preferredBackend (the SDK snapshot option; daemon flag key snapshotPreferredBackend) does override the plan, even under penalty (RunnerTests+SnapshotCapturePlan.swift:243-255), but the CLI snapshot command doesn't expose it (still true in 0.21.28).

Evidence (runner.log)

  • One SNAPSHOT_XCTEST_CHANNEL_PENALIZED … reason=queries_backend_timeout was followed by 70 PRIVATE_AX_SNAPSHOT_USED captures in the next ~2 min.
  • One session's runner.log, covering several days and 43 runner launches, has 273 PENALIZED lines (196 queries_backend_timeout, 72 tree_backend_timeout, 5 slow_tree_capture_…) against 7 PENALTY_CLEARED. All 50 penalties that re-armed inside 120 s are tree_backend_timeout. Most of the queries_backend_timeout ones come shortly after the previous penalty expired (median about 21 s). 41,053 of 57,766 snapshot commands logged XCTEST_CHANNEL_DEFERRED. Another session's log has 115 PENALIZED (60 / 51 / 4) against 1 cleared.

Related: #1542.

Requests

  1. Make the slow threshold and the penalty duration configurable (env or CLI / session option).
  2. Don't penalize the channel for a single 1 s query-sweep timeout alone, and don't let the 1 s recovery probe re-arm the penalty.
  3. Expose preferredBackend on the CLI snapshot command (and as a session default), so a client can force recursiveTree for steps that need an accurate, top-most-screen tree.
  4. Optionally, mark private-AX nodes that belong to a covered (non-top-most) presented screen, so clients can filter them out. (0.21.28's presentedSurface fact, #3565, may cover part of this; not yet tested on our side.)

Happy to provide more logs or test a nightly.

Dominant language
TypeScript
Stars
5.1k
Forks
339
Avg merge
11h 59m
Merged PRs (30d)
736

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from callstack/agent-device

All issues in callstack/agent-device

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.