iOS: XCTest-channel penalty keeps the runner in private-AX fallback most of a long session (fixed 3 s / 120 s, no override)

Read the snapshot penalty and capture plan code under dist/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/, starting with RunnerTests+SnapshotTiming.swift, RunnerTests+SnapshotCapturePlan.swift, and the snapshot CLI entry point. Check the related issue #1542 and presentedSurface work in #3565 before choosing a scope. Done means the requested penalty controls and CLI backend option behave as specified, with tests covering single query timeouts and recovery probes.

Summary

On a heavy UIKit app (iPad simulator), one slow or timed-out XCTest-channel snapshot tier puts the runner into private-AX fallback for 120 s. A timed-out 1 s recursiveTree recovery probe re-arms it; that probe runs when the private-AX capture is rejected. And once the penalty expires, the next capture's 1 s query sweep often times out and penalizes the channel again. In one long-running session log, 71 % of snapshot commands ran with the XCTest channel deferred.

In our captures, private-AX returned presented screens together with the screen underneath them. A test then can't reliably tell which screen is top-most, and an element of the covered screen looks "visible". For read-only checks this is mostly harmless. For steps that must act on the right screen (bulk select / destructive bulk actions in a list), it's unsafe, so we refuse those steps whenever a fallback happened in the last 120 s.

Version / environment

• agent-device 0.21.25 (pinned by @e2e-dev/mobile 0.12.0, used through the e2e 0.20.0 runner). The penalty logic is unchanged in 0.21.28 and 0.21.29-nightly.20261020.
• Xcode 27.0 (27A266a), iOS 27.0 simulator runtime, iPad Air 11-inch (M4) simulators
• UIKit app with long table views

What the source shows (0.21.25, dist/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/)

• An XCTest-channel tier penalizes the channel when it times out, exhausts its deadline, or takes longer than snapshotXCTestSlowCaptureThreshold (3 s): RunnerTests+SnapshotTiming.swift:119-155, constant at RunnerTests.swift:139. Budgets: tree slice 8 s (RunnerTests.swift:143), query sweep 1.0 s (RunnerTests+Snapshot.swift:134-143), viewport read ≤ 1.0 s (RunnerTests+SnapshotAcquisition.swift:50-56). A system-modal probe timeout also penalizes (RunnerTests+Snapshot.swift:360).
• penalizeSnapshotXCTestChannel sets penaltyUntil = now + 120 s (RunnerTests+SnapshotCapturePlan.swift:133-143, RunnerTests.swift:122). While it's active, the plan puts private-AX first, plus recursiveTree with a 1 s slice (RunnerTests+SnapshotCapturePlan.swift:110, 256-277).
• Each slow or timed-out XCTest attempt during the penalty re-arms it. In practice that's the 1 s tree probe, which runs when the private-AX tier fails or is rejected (RunnerTests+SnapshotCapturePlan.swift:272, 391-431). penaltySuppressed applies only to the one-shot warm-up after a target PID change (:304, 373-378; RunnerTests+Lifecycle.swift:254).
• It's cleared only by targetReset / a new target PID (RunnerTests+Lifecycle.swift:228, 252), by the 120 s expiring, or by a runner restart.
• The thresholds and the duration are let constants. No env var, config or CLI flag changes them. preferredBackend (the SDK snapshot option; daemon flag key snapshotPreferredBackend) does override the plan, even under penalty (RunnerTests+SnapshotCapturePlan.swift:243-255), but the CLI snapshot command doesn't expose it (still true in 0.21.28).

Evidence (runner.log)

• One SNAPSHOT_XCTEST_CHANNEL_PENALIZED … reason=queries_backend_timeout was followed by 70 PRIVATE_AX_SNAPSHOT_USED captures in the next ~2 min.
• One session's runner.log, covering several days and 43 runner launches, has 273 PENALIZED lines (196 queries_backend_timeout, 72 tree_backend_timeout, 5 slow_tree_capture_…) against 7 PENALTY_CLEARED. All 50 penalties that re-armed inside 120 s are tree_backend_timeout. Most of the queries_backend_timeout ones come shortly after the previous penalty expired (median about 21 s). 41,053 of 57,766 snapshot commands logged XCTEST_CHANNEL_DEFERRED. Another session's log has 115 PENALIZED (60 / 51 / 4) against 1 cleared.

Related: #1542.

Requests

1. Make the slow threshold and the penalty duration configurable (env or CLI / session option).
2. Don't penalize the channel for a single 1 s query-sweep timeout alone, and don't let the 1 s recovery probe re-arm the penalty.
3. Expose preferredBackend on the CLI snapshot command (and as a session default), so a client can force recursiveTree for steps that need an accurate, top-most-screen tree.
4. Optionally, mark private-AX nodes that belong to a covered (non-top-most) presented screen, so clients can filter them out. (0.21.28's presentedSurface fact, #3565, may cover part of this; not yet tested on our side.)

Happy to provide more logs or test a nightly.