Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

macOS native app backend: follow-ups (guarantees, evidence, persistent helper, session backend)

Aperta
  • Qualcosa non va?
#3,213 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@janicduplessis ci sta già lavorando.

Dal 8/10/2026.

  • #3323 di @janicduplessis — aperta

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
15/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
electron, macos, swift, typescript
Ambito
desktop-dev, testing-qa, tooling

Direzione di ricerca

Start with item 1: read ADR 0031, then trace resolvePressTarget → smallestElement in BackgroundInteraction.swift and the shared guards behind interaction-guarantees.ts. Fixture evidence lands in the macOS host lane in macos.yml (item 3); helper.ts and main.swift are to be split before the item 6 transport work, and MACOS_CLICK_* ↔ MouseClickSchedule.swift is the item 11 pin. Done for the umbrella means items 1–3 merged, each as its own PR naming this issue, with 4–12 done or split out.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement needs-triage

Purpose

Follow-ups for the opt-in macOS native app backend (AGENT_DEVICE_MACOS_APP_BACKEND=native): #3188 (helper app surface), #3189 (routing, admission facts, ADR 0031), #3195 (ghost cursor). The backend works: accessibility actions, no XCTest, the app may stay in the background. These items make it provably correct, evidence-backed, then faster and broader.

The plan was reviewed by Claude and by Codex (codex exec, read-only). The order below is the reconciled one: correctness and evidence come before persistence.

Work items (in order)

  1. Guarantee classification and fallback correctness (M, no deps).

    • Problem: when the hit test names no control, resolvePressTarget → smallestElement (BackgroundInteraction.swift) may act on a different element than the snapshot node the shared guards evaluated. That contradicts ADR 0031's "ADR 0011 guarantees unchanged". Repeated presses also stop early and report fewer clicks without a typed outcome.
    • Required: add a row for this path to interaction-guarantees.ts, and either bound the fallback to the target node's own subtree or report the acted element (role, label, frame, window) so the daemon can verify it. Partial outcomes are typed, never replayed automatically. ADR 0031 is updated.

  2. Snapshot quality and typed AX failures (S–M).

    • Problem: the Chromium warning is free text, and it disappears once the flag is already on without re-checking population. AX attribute errors and permission problems become empty trees.
    • Required: emit a typed SnapshotQualityVerdict (as iOS does), plus typed accessibility and permission failures.

  3. Native fixture lane (M, needs 1–2).

    • Required: a fixture app (AppKit window plus WKWebView) in the macos.yml host lane, run with AGENT_DEVICE_MACOS_APP_BACKEND=native set explicitly.
    • Done when the lane asserts, through production routing and with the app in the background: resulting app state, foreground and real pointer unchanged, typed refusals, and the delivery mechanism per action.
    • Electron coverage is separate: WKWebView does not prove Electron.

  4. Capture-frame contract (M). Snapshots and screenshots carry explicit frame metadata: window identity, origin, scale and display. Native snapshots traverse several windows while the screenshot captures one; XCTest subtracts the window origin at dispatch.

  5. Keys, menu and value actions (S–M, needs 3). Separate increments, each with fixture evidence:

    • key combos via process key events (today's evidence covers only text, Return and Tab);
    • AXShowMenu where a control supports it;
    • increment/decrement for sliders and steppers.

    Not general double-click via AXOpen, and not Cmd+[ as a universal back.

  6. Persistent helper transport (L, needs 3).

    • Measure first, separately: process start, per-invocation source fingerprinting (helper.ts), AX work and cursor time.
    • Then a JSON protocol over stdio following ADR 0002: fresh state per command; cancellation, serialization, and EOF/crash cleanup; bounded waits; behavior across TCC grant/revoke and binary replacement; cursor reset/hide and display changes.
    • Split helper.ts (install / transport / clients) and main.swift before adding behavior.

  7. Element-identity dispatch (L, needs 1 and 6). Act on cached accessibility handles only with helper, app and snapshot-generation tokens, ref-epoch authorization, live validation and bounded retention. Raw x y commands keep hit-testing with window ownership.

  8. Session-scoped backend (L, independent design).

    • open --backend native|xctest, with the env var only as a default.
    • Admission and binding share one immutable execution context: admission inspects only device today, and bindings rebuild device facts.
    • Define sessionless prepare, reopen and recovery semantics.

  9. Native recording (L). A ScreenCaptureKit window stream as its own lifecycle, replacing the runner refusal.

  10. Helper tree presentation (M). Collapse only proven structural wrapper groups (Chromium), keeping raw topology, selector meaning and clip ownership. Use a reproducible fixture, not ad-hoc counts.

  11. Parity cleanup (S).

    • Pin the click-schedule constants (MACOS_CLICK_* ↔ MouseClickSchedule.swift) with a fixture.
    • Keep scroll travel on the shared golden table rather than fetching a frame and sending pixels.

  12. Framework matrix and default decision (L, last). Cover AppKit, SwiftUI, Catalyst, Electron and WebKit, including unsupported actions and sparse apps. Native stays opt-in until this evidence exists.

  13. Pointer events with window fields (M, needs 1 and 3).

    • Evidence (2026-10-05, measurements in the comments): public CGEventPostToPid with the target window number in event field 51, plus field 58 or a window-local location, delivers click, double-click and secondary click to a background AppKit, SwiftUI and WKWebView fixture: 54 of 54, no activation, frontmost app unchanged. A bare post, the ADR 0031 measurement, delivers nothing. Posting through SkyLight adds nothing. Drags and WKWebView button clicks need the private focus-without-raise activation.
    • Required: an event path for double-click, secondary click and points with no accessibility action. Success needs an observed change; otherwise the outcome is a typed unverified result. The path gets its own row in interaction-guarantees.ts and fixture-lane evidence (item 3), including Electron and Catalyst. Fields 51 and 58 are undocumented, so the fixture pins them and a macOS change fails the lane, not users.
    • Not: SkyLight posting or focus-without-raise without a separate decision.

Non-goals

  • A single {facts, interactor, snapshotRoute} facade: the parts have different lifetimes.
  • Turning frontmost-app/menubar presses into accessibility-only presses before their surface guarantees are defined. Synthetic menu extras need pointer delivery.

Completion

Each item lands as its own PR that names this issue. The umbrella closes when items 1–3 have landed, and items 4–13 are either done or split into their own issues.

Lingua principale
TypeScript
Stelle
4.9k
Fork
328
Merge medio
12h 13m
PR unite (30g)
541

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di callstack/agent-device

Tutte le issue di callstack/agent-device

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.