Computer Use (Windows/UIA): snapshot element list is flattened in reverse pre-order; ToggleState not exposed
Maintainer antworten meist innerhalb von 1 Tag
Dieses Issue hat noch niemand übernommen.
Bewertung
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Anfängerfreundlichkeit
- 49/100
Rechercherichtung
Start by tracing the Windows computer-use snapshot code that builds the flat element listing and renders element attributes; the issue does not name those files or tests. Check how traversal order is produced and which UIA properties are included. Done means the listing follows forward pre-order (or documents its order) and exposes applicable state attributes, including ToggleState; run the relevant snapshot tests.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Environment
- ZCode computer-use plugin 0.6.3 (official), Windows 10 x64 (19045)
- App under test: a GPUI (gpui-kit 0.7.0) Todo list on Windows, observed via the
UIA accessibility path (get_app_state/getAXState)
Summary
Two consumer-side issues in the Windows computer-use snapshot element listing:
-
Element order is reverse pre-order. The flat element list enumerates the
UIA tree like a stack-based DFS that emits a node before pushing its
children in order — equivalent to a DFS pre-order with every sibling group
reversed. Flat indices therefore do not match visual order: within each
container the list runs bottom-row → top-row. We verified this across four
distinct container levels in one app (table rows, a toolbar, a form row, an
inline edit card). -
ToggleState is missing from the listing. Checkbox elements show only
checkbox (pressable) actions=[AXToggle,AXPress]with no state field, even
though the element advertises the toggle action — which in UIA implies the
Toggle pattern is registered andUIA_ToggleToggleStatePropertyIdis
readable. (By contrast, textfieldvalueis shown, so attribute display
is selective rather than absent.)
Why we're confident it's the snapshot layer, not the app toolkit
We verified the app's whole a11y chain is strictly order-preserving and
state-reporting:
- gpui builds accesskit children in document order during prepaint DFS
(window/a11y.rs:429-440,element.rs:367-406); NodeId is identity-only. - accesskit 0.24.1
Node.childrenis an ordered Vec; accesskit_consumer 0.38.0
iterates it unchanged; accesskit_windows 0.34.0Navigateuses
filtered_childrenorder (the filter drops, never reorders). - accesskit_windows registers TogglePattern with a proper ToggleState for
checkboxes (node.rs:575-585,1332-1335). - Windows Inspect.exe shows the same tree in visual order with readable
ToggleState.
Replaying 12 captured snapshots against the reverse-pre-order model reproduced
the emitted listing character-for-character, including which elements
appear before/after others in every container. Tree-shape changes (adding a
row, opening an edit card) explain all frame-to-frame index shifts.
Impact
An agent clicking by flat index hits a visually different row than intended
(observed: "click row 2 by index" landed on visual row 3 — the click used the
element's own bounds, so it struck the element the index actually named).
Checkbox verification cannot be asserted from the snapshot at all.
Suggested fixes
- Flatten in forward pre-order (FirstChild → NextSibling semantics), or at
minimum document that flat order is reverse pre-order. - Include state attributes in element lines: ToggleState (On/Off), Value,
IsSelected where applicable.
- Vorherrschende Sprache
- Python
- Sterne
- 86
- Forks
- 51
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Entwicklungsumgebung
- Kein Dockerfile und keine Docker-Compose-Datei
- Hat eine Pull-Request-Vorlage
- Beitragsleitfaden lesen
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus zai-org/zcode-plugins
-
video-agent-kit (0.4.3): hooks invoke python3, which doesn't exist on Windows — SessionStart/UserPromptSubmit/Stop hooks fail on every triggerEvtl. vergeben @dvd233 hat das vor 14 Tagen übernommen. Offen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 85/100
zai-org/zcode-plugins#14 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 4/5 3-5 Tage Anfängerfreundlichkeit 52/100
zai-org/zcode-plugins#64 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 5/5 Über eine Woche Anfängerfreundlichkeit 25/100
zai-org/zcode-plugins#63 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 3/5 1-2 Tage Anfängerfreundlichkeit 25/100
zai-org/zcode-plugins#60 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 4/5 3-5 Tage Anfängerfreundlichkeit 38/100
zai-org/zcode-plugins#59 ·
Maintainer antworten meist innerhalb von 1 Tag
Alle Issues in zai-org/zcode-plugins
Ähnliche Issues
-
Schwierigkeit 1/5 1-3 Stunden Anfängerfreundlichkeit 85/100
pytest-dev/pluggy#757 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 1/5 1-3 Stunden Anfängerfreundlichkeit 85/100
NousResearch/hermes-agent#134960 ·
Maintainer antworten meist innerhalb von 1 Tag
-
HTML backend: `<br>` leaks the internal sentinel U+E000 into list items, headings and captionsEvtl. vergeben @morten-lagabote hat das heute übernommen. Offen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 67/100
docling-project/docling#4671 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 70/100
Maintainer antworten meist innerhalb von 1 Tag
-
good first issue hacktoberfest infra
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 78/100
Maintainer antworten meist innerhalb von 1 Tag