Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

macOS native app backend: follow-ups (guarantees, evidence, persistent helper, session backend)

Ouverte
  • Quelque chose ne va pas ?
#3,213 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@janicduplessis y travaille déjà.

Depuis le 26/10/2026.

  • #3962 par @janicduplessis — ouverte
  • #3965 par @janicduplessis — ouverte

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
15/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
electron, macos, swift, typescript
Domaine
desktop-dev, testing-qa, tooling

Piste de recherche

Start with item 1: read ADR 0031, then trace resolvePressTarget → smallestElement in BackgroundInteraction.swift and the shared guards behind interaction-guarantees.ts. Fixture evidence lands in the macOS host lane in macos.yml (item 3); helper.ts and main.swift are to be split before the item 6 transport work, and MACOS_CLICK_* ↔ MouseClickSchedule.swift is the item 11 pin. Done for the umbrella means items 1–3 merged, each as its own PR naming this issue, with 4–12 done or split out.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

backlog enhancement

Purpose

Follow-ups for the opt-in macOS native app backend (AGENT_DEVICE_MACOS_APP_BACKEND=native): #3188 (helper app surface), #3189 (routing, admission facts, ADR 0031), #3195 (ghost cursor). The backend works: accessibility actions, no XCTest, the app may stay in the background. These items make it provably correct, evidence-backed, then faster and broader.

The plan was reviewed by Claude and by Codex (codex exec, read-only). The order below is the reconciled one: correctness and evidence come before persistence.

Work items (in order)

  1. Guarantee classification and fallback correctness (M, no deps).

    • Problem: when the hit test names no control, resolvePressTarget → smallestElement (BackgroundInteraction.swift) may act on a different element than the snapshot node the shared guards evaluated. That contradicts ADR 0031's "ADR 0011 guarantees unchanged". Repeated presses also stop early and report fewer clicks without a typed outcome.
    • Required: add a row for this path to interaction-guarantees.ts, and either bound the fallback to the target node's own subtree or report the acted element (role, label, frame, window) so the daemon can verify it. Partial outcomes are typed, never replayed automatically. ADR 0031 is updated.

  2. Snapshot quality and typed AX failures (S–M).

    • Problem: the Chromium warning is free text, and it disappears once the flag is already on without re-checking population. AX attribute errors and permission problems become empty trees.
    • Required: emit a typed SnapshotQualityVerdict (as iOS does), plus typed accessibility and permission failures.

  3. Native fixture lane (M, needs 1–2).

    • Required: a fixture app (AppKit window plus WKWebView) in the macos.yml host lane, run with AGENT_DEVICE_MACOS_APP_BACKEND=native set explicitly.
    • Done when the lane asserts, through production routing and with the app in the background: resulting app state, foreground and real pointer unchanged, typed refusals, and the delivery mechanism per action.
    • Electron coverage is separate: WKWebView does not prove Electron.

  4. Capture-frame contract (M). Snapshots and screenshots carry explicit frame metadata: window identity, origin, scale and display. Native snapshots traverse several windows while the screenshot captures one; XCTest subtracts the window origin at dispatch.

  5. Keys, menu and value actions (S–M, needs 3). Separate increments, each with fixture evidence:

    • key combos via process key events (today's evidence covers only text, Return and Tab);
    • AXShowMenu where a control supports it;
    • increment/decrement for sliders and steppers.

    Not general double-click via AXOpen, and not Cmd+[ as a universal back.

  6. Persistent helper transport (L, needs 3).

    • Measure first, separately: process start, per-invocation source fingerprinting (helper.ts), AX work and cursor time.
    • Then a JSON protocol over stdio following ADR 0002: fresh state per command; cancellation, serialization, and EOF/crash cleanup; bounded waits; behavior across TCC grant/revoke and binary replacement; cursor reset/hide and display changes.
    • Split helper.ts (install / transport / clients) and main.swift before adding behavior.

  7. Element-identity dispatch (L, needs 1 and 6). Act on cached accessibility handles only with helper, app and snapshot-generation tokens, ref-epoch authorization, live validation and bounded retention. Raw x y commands keep hit-testing with window ownership.

  8. Session-scoped backend (L, independent design).

    • open --backend native|xctest, with the env var only as a default.
    • Admission and binding share one immutable execution context: admission inspects only device today, and bindings rebuild device facts.
    • Define sessionless prepare, reopen and recovery semantics.

  9. Native recording (L). A ScreenCaptureKit window stream as its own lifecycle, replacing the runner refusal.

  10. Helper tree presentation (M). Collapse only proven structural wrapper groups (Chromium), keeping raw topology, selector meaning and clip ownership. Use a reproducible fixture, not ad-hoc counts.

  11. Parity cleanup (S).

    • Pin the click-schedule constants (MACOS_CLICK_* ↔ MouseClickSchedule.swift) with a fixture.
    • Keep scroll travel on the shared golden table rather than fetching a frame and sending pixels.

  12. Framework matrix and default decision (L, last). Cover AppKit, SwiftUI, Catalyst, Electron and WebKit, including unsupported actions and sparse apps. Native stays opt-in until this evidence exists.

  13. Pointer events with window fields (M, needs 1 and 3).

    • Evidence (2026-10-05, measurements in the comments): public CGEventPostToPid with the target window number in event field 51, plus field 58 or a window-local location, delivers click, double-click and secondary click to a background AppKit, SwiftUI and WKWebView fixture: 54 of 54, no activation, frontmost app unchanged. A bare post, the ADR 0031 measurement, delivers nothing. Posting through SkyLight adds nothing. Drags and WKWebView button clicks need the private focus-without-raise activation.
    • Required: an event path for double-click, secondary click and points with no accessibility action. Success needs an observed change; otherwise the outcome is a typed unverified result. The path gets its own row in interaction-guarantees.ts and fixture-lane evidence (item 3), including Electron and Catalyst. Fields 51 and 58 are undocumented, so the fixture pins them and a macOS change fails the lane, not users.
    • Not: SkyLight posting or focus-without-raise without a separate decision.

Non-goals

  • A single {facts, interactor, snapshotRoute} facade: the parts have different lifetimes.
  • Turning frontmost-app/menubar presses into accessibility-only presses before their surface guarantees are defined. Synthetic menu extras need pointer delivery.

Completion

Each item lands as its own PR that names this issue. The umbrella closes when items 1–3 have landed, and items 4–13 are either done or split into their own issues.

Langage dominant
TypeScript
Étoiles
5.2k
Forks
359
Merge moyen
12 h 48 min
PR mergées (30 j)
657

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de callstack/agent-device

Toutes les issues de callstack/agent-device

Issues similaires

Plus d'issues TypeScript

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.