refactor(apple-runner): classify AX snapshot failure once at acquisition
Les mainteneurs répondent en général sous 1 jour
Personne n'a encore pris cette issue.
Évaluation
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Accessibilité débutants
- 42/100
- Type d'issue
- Refactorisation
- Clarté
- Clairement spécifiée
- Activité
- Active
- Stack technique
- swift
- Domaine
- mobile-dev, testing-qa
Piste de recherche
Start by inventorying SnapshotCaptureFailure constructors and isAxSnapshotFailure consumers in apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/, especially RunnerTests+SnapshotAcquisition.swift, RunnerTests+SnapshotCapturePlan.swift, and RunnerTests+SnapshotPresentationInvariant.swift. Extend the named UnitTests snapshot, capture-plan, and presentation-invariant suites; done means downstream classification is code-only, unrelated AX-like prose stays unrelated, and recognized native failures preserve recovery behavior.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Description
Classify native AX snapshot failure once at acquisition
Deletion-first acceptance
This is part of #2803's simplify-and-shrink initiative. Before production simplification, name the production mechanism, duplicated decision, state, fallback, forwarding layer, or public/internal interface this change removes. Compare deletion and inlining against extraction. Moving the same state into another file is not a deletion case.
Every PR reports baseline/final SHAs; added, deleted and net production lines; test/fixture/documentation changes separately; and the exact mechanisms removed. Use a rename-aware diff (git diff --numstat -M BASE...HEAD) and account for all affected production paths, including destination packages. Moving tests out of production files, generated output, formatting churn, and moving code elsewhere are not production-code deletion. Preserve readable code; no compressed formatting to hit a number.
Default for simplification work: net production reduction. A required behavior-preserving size split may be line-neutral. Growth needs an explicit, quantified explanation of the correctness or type guarantee it buys, why deletion/inlining cannot achieve it, and what old mechanism is removed; it is not automatically acceptable because ownership looks cleaner. If the only benefit is relocating fields or adding a wrapper, defer the change. Required regression tests and a minimal correctness fix are allowed to grow without inventing unrelated deletion to offset them.
Priority and scope gate
Early small deletion: remove downstream AX message classification while retaining necessary native-edge normalization. Prefer deleting the redundant branch/helper over introducing a new error hierarchy or classifier abstraction. Report its production delta separately from added negative regression cases. Serialize shared runner Swift edits; #2911 is already merged.
Problem and evidence
The snapshot acquisition edge already turns a recognized AX serialization failure into SnapshotCaptureFailure(code: axSnapshotErrorCode, ...). Downstream policy nevertheless classifies it again from the human message. That allows wording on an unrelated failure to affect recovery, the quality reason, and terminal target invalidation.
Verified against origin/main at 5712a3552e209c9c853bc1f42cb3e007f1213139: apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotAcquisition.swift:116–134 performs native classification, :159–181 constructs the typed failure and then permits message-based reclassification. RunnerTests+SnapshotCapturePlan.swift:361 retains AX failure for terminal policy, and RunnerTests+SnapshotPresentationInvariant.swift:17 derives its quality reason. The latest placeholder changes do not change these sites.
Intended design
Keep unavoidable private-API text interpretation at the native acquisition edge. Above that edge, a snapshot failure's machine code is the sole AX classification authority. Preserve the current recognized native messages, public code, diagnostic message, hint, raw-snapshot failure behavior, and interactive fail-closed behavior.
The intended interface is deliberately small:
static func isAxSnapshotFailure(_ failure: SnapshotCaptureFailure) -> Bool {
failure.code == axSnapshotErrorCode
}
Use the existing failure carrier; do not add a second isAXFailure flag that can disagree with its code. Keep or name a private acquisition-edge classifier for actual native error/exception evidence. If NSError or NSException exposes reliable structured evidence, characterize it first and prefer it only where proven. This issue does not depend on discovering a new native code or eliminating all native-edge text parsing.
Implementation and acceptance
- Inventory every constructor of
SnapshotCaptureFailureand every consumer ofisAxSnapshotFailure, including private-AX and presentation error mapping. Prove each genuine AX path reaches the same machine code before tightening consumers. - Add paired behavioral regressions through existing product seams: a typed AX failure with neutral wording remains AX; an unrelated typed failure containing identical AX-looking prose remains unrelated. Assert both quality classification and the terminal capture-plan consequence.
- Make downstream classification code-only. If a producer currently relies on downstream text classification, move that normalization to its actual acquisition edge while retaining its observed classification and public error details.
- Retain native exception handling, source diagnostic text, and the capture-plan ordering. Keep tests with the owning modules; do not add test-only production exports.
Before/after measures: there is no downstream AX message sniff after normalization; all previously recognized native AX failures keep their code and recovery result; changing diagnostic prose cannot alter quality or terminal policy. Plant the old downstream text fallback and observe the unrelated-code negative fail. A native-edge fixture still recognizes each currently supported signature, including the closest negative that resembles an AX message but lacks the full recognized signature.
Validation
Extend the owning snapshot acquisition/quality/terminal tests, including UnitTests/RunnerTests+SnapshotTests.swift, RunnerTests+SnapshotCapturePlanTests.swift, and RunnerTests+SnapshotPresentationInvariantTests.swift as appropriate to the actual call sites. Pure classification belongs in the macOS host XCTest lane; native exception capture semantics belong in the iOS simulator lane. Preserve RunnerTests+SnapshotCapturePlanOccupancyTests.swift as integration proof of recovery while abandoned native work remains outstanding.
Run pnpm check:xctest-selection, pnpm build:xcuitest:ios, and pnpm build:xcuitest:macos; run the selected host/simulator XCTest cases through the repository's existing harness and record the selected cases. Finish with pnpm check:affected --run and all reported exact-head obligations. Do not substitute compilation for XCTest evidence or claim a real-device AX failure was reproduced unless measured.
Scope and readiness
Ready for implementation after the constructor inventory; unexpected producer classification must be characterized before changing it. Independent child of umbrella #2803, with no dependency on broader capture-plan extraction. No retry, timeout, fallback order, or error-message cleanup changes. Wire-schema work is limited to the typed fallback disclosure and consuming assertion consolidated from #3328 below. Expected effort: S–M; risk: medium because AX classification controls fail-closed behavior.
Source paths in this issue are rooted at apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/; extension names abbreviate the RunnerTests+ prefix where stated. Tests live in its UnitTests/ directory. For a fresh worktree, run pnpm install --frozen-lockfile && pnpm build before the focused checks. Read docs/agents/testing.md and docs/agents/device-verification.md for native lane selection and build freshness.
Consolidated acceptance from #3328
The regular depth-1 simulator snapshot may disclose a legitimate XCTest fallback when the AX bridge probe circuit is disabled. Preserve the fallback disclosure. Classify the acquisition decision once and expose its typed fallback reason to the consuming smoke assertion; do not classify warning strings.
- Carry the typed acquisition/fallback reason through the existing snapshot quality response contract.
- Update
test/integration/ios-simulator-e2e/live-snapshot-depth-frontier.tsto assert the disclosed fallback through that typed reason and quality state. - Retain negative coverage so an unrelated failure with AX-looking prose cannot select the fallback or its quality classification.
- Keep the historical signatures owned here in #2491's lane ledger.
The live-run leg from #3328 is already satisfied: run 37843633972 at 98321a2 passed smoke:regular-visible-depth-frontier with healthy/tree plus circuit-disabled disclosure. This is evidence for that head, not proof that a future implementation passed. The remaining work is typed wire disclosure and the assertion. #3328 closes as consolidated into this issue.
- Langage dominant
- TypeScript
- Étoiles
- 5.1k
- Forks
- 339
- Merge moyen
- 12 h 50 min
- PR mergées (30 j)
- 632
Préparer son environnement
- Aucun Dockerfile ni fichier Docker Compose
- Aucun modèle de pull request
- Lire le guide de contribution
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Autres issues de callstack/agent-device
-
Difficulté 2/5 1-3 heures Accessibilité débutants 64/100
callstack/agent-device#3945 ·
Les mainteneurs répondent en général sous 1 jour
-
Limrun iOS fill reports success but drops non-ASCII textPeut-être pris @mvanhorn l’a pris il y a 5 jours. Ouvertebug ready-for-agent
Difficulté 2/5 1-3 heures Accessibilité débutants 87/100
callstack/agent-device#3819 · 2 commentaires ·
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 68/100
callstack/agent-device#1869 ·
Les mainteneurs répondent en général sous 1 jour
-
iOS keyboard capture can race dismissal and omit the input assistant barPeut-être pris @shamilovtim l’a pris aujourd’hui. Ouverte
Difficulté 4/5 3-5 jours Accessibilité débutants 51/100
callstack/agent-device#3957 ·
Les mainteneurs répondent en général sous 1 jour
-
iOS fill can report TEXT_ENTRY_MISMATCH before the confirmation deadlinePeut-être pris @shamilovtim l’a pris aujourd’hui. Ouverte
Difficulté 3/5 1-2 jours Accessibilité débutants 35/100
callstack/agent-device#3952 ·
Les mainteneurs répondent en général sous 1 jour
Toutes les issues de callstack/agent-device
Issues similaires
-
bug
Difficulté 2/5 1-3 heures Accessibilité débutants 76/100
awslabs/visual-asset-management-system#804 ·
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 70/100
restatedev/sdk-typescript#813 ·
-
bug desktop P1 session XS
Difficulté 2/5 1-3 heures Accessibilité débutants 82/100
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 62/100
Merit-Systems/x402scan#1312 ·
-
type:bug
Difficulté 2/5 1-3 heures Accessibilité débutants 78/100
Azure/apiops-cli#331 ·
Les mainteneurs répondent en général sous 3 jours