Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Agent task coverage audit

Ouverte
  • Quelque chose ne va pas ?
#75 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
35/100
Type d'issue
Documentation
Clarté
Clairement spécifiée
Activité
Active
Stack technique
swift
Domaine
desktop, devtools

Piste de recherche

Start by reading docs/tools.md, docs/recipes.md, src/tools.ts, and the native client, then compare the documented task sequences with the current code and the referenced issues and pull requests. The audit says its ratings were not checked live and points to #42 and npm run live-check for pending verification. Done means updating the task coverage assessment with evidence for each rating and recording any remaining gaps.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

documentation

How well desktop-tools covers everyday agent tasks on a Mac, mapped from docs/tools.md, docs/recipes.md, src/tools.ts and the native client.

Refreshed 2026-10-17 (second pass) against main at aefabc2 (0.11.0 with #107 and #109), plus open #110 (web accessibility for desktop_wait) and #111 (incomplete-read retry). The first refresh was at 6783905 (0.8.0); the original audit was at acae444. Rows changed since then say which PR changed them. None of this was run live: the screen is locked, and the live checks for each PR are in #42. Ratings come from the code, and the ones marked (unverified) depend on how an app or the OS exposes its UI.

  • Gap: there is no reasonable tool sequence.
  • Awkward: it works, but takes many calls, has an unverifiable result, or relies on a fragile workaround.
  • Works: there is a direct sequence.

Common shorthand: inspect = desktop_inspect (it returns the snapshot that input uses). Each completed action returns a fresh inspection, so chains don't need a separate inspect.

Counts: 5 gaps (10 originally), 11 awkward (13 at the first refresh, 16 originally), 29 works (26 at the first refresh, 18 originally).

Gaps (5)

# Task Tool sequence Why it's a gap
2 Enter a password: a website login, joining Wi-Fi, or an admin prompt (SecurityAgent) inspect → desktop_type / desktop_insert / desktop_dialog_answer field All three refuse secure fields, and so does #82's web typing. Left for the user's decision.
3 Install an app from a DMG when it replaces an existing app, or change a locked System Settings pane desktop_open dmg → inspect → desktop_drag app → Applications The copy works, but admin authentication hits gap 2. Touch ID can't be done at all.
4 Connect to a secured Wi-Fi network desktop_menubar_click Wi-Fi → inspect → desktop_click network → password Opening the list and choosing the network work, and an open network works. The password field hits gap 2.
8 Move a window to another Space — Documented as not supported: since macOS 14.5 only the Dock can do it. The Mission Control drag workaround is always unknown.
10 Select non-editable text in a PDF (to copy or highlight) inspect → desktop_drag Web pages are done (row g10 under Works). A PDF view has no selected text range, and #88's WebKit text markers apply to web areas only, so a PDF needs a drag-select, which is unverifiable. desktop_read covers copying.

Works but awkward (11)

# Task Tool sequence Why awkward
2 Use Spotlight to open something desktop_system_key Cmd-Space with expect window_appears → inspect → desktop_insert → desktop_key enter Many steps. The system key now completes when Spotlight's window appears (expect, #89), but is still unknown otherwise. #84 has a recipe. desktop_open / desktop_launch is usually better.
4 Print to PDF desktop_menu File > Print… → … → desktop_dialog_answer About 6 supported steps. #84 has a recipe, including the Export as PDF shortcut.
7 Drag a file into a web upload area Finder inspect + browser inspect → desktop_drag with to_snapshot Cross-app drags are foreground and unknown unless an expect condition (#89), such as the file name appearing, holds afterwards. Prefer the upload button and desktop_dialog_answer path.
8 Install an app from a DMG (no replace) desktop_open dmg → desktop_wait → inspect → desktop_drag → desktop_dialog Gatekeeper Many steps with waits. #84 has a recipe.
9 Use the share sheet desktop_menu File > Share > , or the toolbar button → popover The menu path works. The popover may live in another process (unverified). #84 has a recipe.
11 Pick a color in the color panel desktop_menu Format > Show Colors → … → desktop_type hex → desktop_key enter Many steps, and the panel layout varies (unverified). #84 has a recipe.
12 Choose a font desktop_menu Format > Font > Show Fonts → … The lists are long. #86's table read can list them. #84 has a recipe.
14 Copy files between Finder folders desktop_clipboard_write files → inspect target → desktop_key Cmd-V Verifiable but indirect. #99's desktop_insert format: "files" pastes into text areas only; a Finder folder paste confirmed by the copied names is a #3 follow-up. #84 has a recipe. Shell cp is simpler.
15 Respond to a notification desktop_notifications → desktop_notification_action Built in #74 against macOS 11–15. Pending a live check on 27 (#42).
16 Use a context menu on a Dock item desktop_dock right-click + item Moves the real pointer.
7g Zoom or rotate in Maps, Photos or Preview (was gap 7) desktop_menus → desktop_menu View > Zoom In / Rotate, or desktop_key Cmd-= / Cmd-- (with repeat) Use the menu or keys (#87 recipe). There is no pinch or rotate gesture by decision: undocumented events, GPL prior art, and no Chrome support. Canvas views may not report the zoom level.

Works (29)

# Task Tool sequence
1 Rename a file in Finder inspect → desktop_click file → desktop_key enter → desktop_type → desktop_key enter
2 Attach a file in Mail desktop_menu File > Attach Files… → desktop_dialog_answer path → button Choose File
3 Change a System Settings toggle desktop_open settings URL → desktop_wait element_appears → desktop_set_value with readback
4 Accept a cookie banner inspect (with scope if truncated) → desktop_click Accept
5 Use a context menu desktop_click kind right or AXShowMenu → AXPick
6 Scroll a list until an item is visible AXScrollToVisible, or desktop_scroll + desktop_wait element_appears
7 Switch tabs in Safari or Terminal desktop_click tab, desktop_menu, or desktop_key Cmd-Shift-]
8 Answer a permission prompt (button) desktop_dialog → desktop_dialog_answer button
9 Choose from a popup button or web <select> desktop_click → AXPick
10 Handle a modal sheet or alert desktop_wait dialog_appears → desktop_dialog → desktop_dialog_answer
11 Search the web desktop_open URL → desktop_wait text_contains → desktop_read. In Chrome and other Chromium browsers, the page's text appears once web accessibility is on (#107 for inspect and read, #110 for the wait).
12 Read a long article or mail thread desktop_read
13 Save a document to a given path desktop_menu File > Save… → desktop_dialog_answer path → Save
14 Quit an app with unsaved changes desktop_quit → desktop_dialog → desktop_dialog_answer button
15 Copy and paste between apps Cmd-C → desktop_clipboard_read, or desktop_clipboard_write → desktop_insert
16 Run a Terminal command and read its output desktop_insert → Enter → desktop_wait text_changes → desktop_read
17 Arrange windows: display, size, full screen desktop_spaces → desktop_move / desktop_resize / desktop_maximize, and desktop_fullscreen with an explicit enabled (#78)
18 Reorder items by dragging within one window desktop_drag (background, exact window)
g1 Select several files or messages (Cmd-click, Shift-click) (was gap 1) desktop_click with modifiers command/shift/option, through the foreground modifier route (#77). The result is unknown unless an expect condition holds afterwards (#89); otherwise check the selection by inspecting.
g5 Set a date or time in a date picker (was gap 5) desktop_set_value with an ISO 8601 string on AXDateTimeArea/AXDateField/AXTimeField, read back. The keyboard route is given when a picker can't be set (#79). Web <input type=date> is (unverified).
g6 Take a screenshot to a file (was gap 6) desktop_screen or desktop_inspect with save_path, region and overwrite, writing the full-resolution PNG (#81)
g9 Volume, brightness or play/pause by key (was gap 9) desktop_system_key volume_up, mute, play_pause, brightness_up… The result is unknown (#80), and these keys have no observable condition for expect. For a set level, use the Control Center slider.
a1 Keyboard-only navigation (was awkward 1) desktop_key with keys (up to 16 chords) or repeat (up to 32) against one snapshot, focus re-checked before each key (#77)
a3 Fill a web form's text fields (was awkward 3) desktop_click the field → desktop_type, which types with key events so page scripts see input, then reads the field back (#82, recipe in #84). In Chromium and Electron apps (Chrome, Slack, VS Code), the fields are listed once #107 turns on the app's web accessibility at the first inspect.
a5 Read a table: Numbers, Finder list, web table (was awkward 5) desktop_read with table: true and a scope → columns, rows of cells, outline depth, and row IDs for desktop_click (#86)
a13 Adjust volume or brightness to a level (was awkward 13) desktop_menubar_click Control Center → desktop_set_value slider with readback (recipe in #84), or the keys in g9
a6 Edit a spreadsheet or table cell (was awkward 6) desktop_read with table: true → desktop_set_cell by row and column, confirmed by reading the table again (#92). Whether Numbers and Excel cells expose a control is (unverified), listed in #42; a cell without one is refused with CELL_NOT_EDITABLE.
a10 Switch the input source (was awkward 10) desktop_input_source with select, read back through the host, covered by pause and the activity list (#90, #101)
g10 Select non-editable text on a web page (was part of gap 10) desktop_select on an element inside a web area, through WebKit text markers, read back (#88). Chromium pages are (unverified).

Top remaining gaps and awkward items

Password entry (gap 2, and gaps 3 and 4 behind it) is left for the user's decision. The first refresh's top three are done: table cells (#92), input source by ID (#90, #101) and expect for foreground input (#89).

  1. Run the pending live checks in #42. None of these ratings were run live, and most rows since 0.5.0 rest on code alone. npm run live-check (#102) automates part of them.
  2. Background paste into web content (#5 part a). It affects a3 and Ace's composer when the app isn't frontmost. A live spike, listed in #42, must answer its three questions before any patch.
  3. Paste files into a Finder folder (awkward 14, a #3 follow-up). Confirm the paste by the copied names appearing in the folder, so the copy is completed rather than indirect.
  4. PDF text selection (gap 10). First find out what Preview's PDF view exposes to Accessibility.
Langage dominant
Swift
Étoiles
0
Forks
0
Merge moyen
15 min
PR mergées (30 j)
94

Préparer son environnement

Ce projet ne fournit ni conteneur de développement, ni Dockerfile, ni guide de contribution : l'installation est à votre charge. Commencez par son README, et consultez notre guide de la première contribution pour les étapes générales.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de githubnext/desktop-tools

Toutes les issues de githubnext/desktop-tools

Issues similaires

Plus d'issues Swift

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.