iOS: XCTest-channel penalty keeps the runner in private-AX fallback most of a long session (fixed 3 s / 120 s, no override)

Geschlossen
  • Stimmt etwas nicht?
#3,844 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
35/100
Issue-Typ
Feature
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
ios, typescript
Bereich
mobile, performance, testing

Rechercherichtung

Read the snapshot penalty and capture plan code under dist/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/, starting with RunnerTests+SnapshotTiming.swift, RunnerTests+SnapshotCapturePlan.swift, and the snapshot CLI entry point. Check the related issue #1542 and presentedSurface work in #3565 before choosing a scope. Done means the requested penalty controls and CLI backend option behave as specified, with tests covering single query timeouts and recovery probes.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

bug needs-info

Summary

On a heavy UIKit app (iPad simulator), one slow or timed-out XCTest-channel snapshot tier puts the runner into private-AX fallback for 120 s. A timed-out 1 s recursiveTree recovery probe re-arms it; that probe runs when the private-AX capture is rejected. And once the penalty expires, the next capture's 1 s query sweep often times out and penalizes the channel again. In one long-running session log, 71 % of snapshot commands ran with the XCTest channel deferred.

In our captures, private-AX returned presented screens together with the screen underneath them. A test then can't reliably tell which screen is top-most, and an element of the covered screen looks "visible". For read-only checks this is mostly harmless. For steps that must act on the right screen (bulk select / destructive bulk actions in a list), it's unsafe, so we refuse those steps whenever a fallback happened in the last 120 s.

Version / environment

  • agent-device 0.21.25 (pinned by @e2e-dev/mobile 0.12.0, used through the e2e 0.20.0 runner). The penalty logic is unchanged in 0.21.28 and 0.21.29-nightly.20261020.
  • Xcode 27.0 (27A266a), iOS 27.0 simulator runtime, iPad Air 11-inch (M4) simulators
  • UIKit app with long table views

What the source shows (0.21.25, dist/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/)

  • An XCTest-channel tier penalizes the channel when it times out, exhausts its deadline, or takes longer than snapshotXCTestSlowCaptureThreshold (3 s): RunnerTests+SnapshotTiming.swift:119-155, constant at RunnerTests.swift:139. Budgets: tree slice 8 s (RunnerTests.swift:143), query sweep 1.0 s (RunnerTests+Snapshot.swift:134-143), viewport read ≤ 1.0 s (RunnerTests+SnapshotAcquisition.swift:50-56). A system-modal probe timeout also penalizes (RunnerTests+Snapshot.swift:360).
  • penalizeSnapshotXCTestChannel sets penaltyUntil = now + 120 s (RunnerTests+SnapshotCapturePlan.swift:133-143, RunnerTests.swift:122). While it's active, the plan puts private-AX first, plus recursiveTree with a 1 s slice (RunnerTests+SnapshotCapturePlan.swift:110, 256-277).
  • Each slow or timed-out XCTest attempt during the penalty re-arms it. In practice that's the 1 s tree probe, which runs when the private-AX tier fails or is rejected (RunnerTests+SnapshotCapturePlan.swift:272, 391-431). penaltySuppressed applies only to the one-shot warm-up after a target PID change (:304, 373-378; RunnerTests+Lifecycle.swift:254).
  • It's cleared only by targetReset / a new target PID (RunnerTests+Lifecycle.swift:228, 252), by the 120 s expiring, or by a runner restart.
  • The thresholds and the duration are let constants. No env var, config or CLI flag changes them. preferredBackend (the SDK snapshot option; daemon flag key snapshotPreferredBackend) does override the plan, even under penalty (RunnerTests+SnapshotCapturePlan.swift:243-255), but the CLI snapshot command doesn't expose it (still true in 0.21.28).

Evidence (runner.log)

  • One SNAPSHOT_XCTEST_CHANNEL_PENALIZED … reason=queries_backend_timeout was followed by 70 PRIVATE_AX_SNAPSHOT_USED captures in the next ~2 min.
  • One session's runner.log, covering several days and 43 runner launches, has 273 PENALIZED lines (196 queries_backend_timeout, 72 tree_backend_timeout, 5 slow_tree_capture_…) against 7 PENALTY_CLEARED. All 50 penalties that re-armed inside 120 s are tree_backend_timeout. Most of the queries_backend_timeout ones come shortly after the previous penalty expired (median about 21 s). 41,053 of 57,766 snapshot commands logged XCTEST_CHANNEL_DEFERRED. Another session's log has 115 PENALIZED (60 / 51 / 4) against 1 cleared.

Related: #1542.

Requests

  1. Make the slow threshold and the penalty duration configurable (env or CLI / session option).
  2. Don't penalize the channel for a single 1 s query-sweep timeout alone, and don't let the 1 s recovery probe re-arm the penalty.
  3. Expose preferredBackend on the CLI snapshot command (and as a session default), so a client can force recursiveTree for steps that need an accurate, top-most-screen tree.
  4. Optionally, mark private-AX nodes that belong to a covered (non-top-most) presented screen, so clients can filter them out. (0.21.28's presentedSurface fact, #3565, may cover part of this; not yet tested on our side.)

Happy to provide more logs or test a nightly.

Vorherrschende Sprache
TypeScript
Sterne
5.3k
Forks
377
Ø Merge
13 Std. 58 Min.
Gemergte PRs (30 T.)
654

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus callstack/agent-device

Alle Issues in callstack/agent-device

Ähnliche Issues

Weitere Issues zu TypeScript

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.