macOS: SubclassingAdapter hides the view's native subviews, such as an embedded WKWebView, from assistive technology

未关闭
  • 哪里不对？
#859 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
15/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
活跃
技术栈
macos, objective-c, rust
领域
accessibility, desktop, operating-systems

调研方向

The relevant code is in adapters/macos/src/subclass.rs, in the children and hit_test Objective-C callbacks and the proposed native_subviews and native_hit_test helpers. The issue asks a design question first: whether a PR that exposes native subviews beside the AccessKit nodes is wanted, or whether a node should stand for a native view in reading order. Wait for a maintainer decision on that shape before writing code. Done means that decision is made and a macOS accessibility dump shows embedded content reachable by VoiceOver.

由索引模型根据 Issue 内容生成。

描述

Problem

SubclassingAdapter replaces the view's accessibilityChildren and accessibilityHitTest: with the AccessKit tree. A native subview the app adds to that view drops out of the accessibility tree. In our case it is a WKWebView showing an HTML mail body inside a GPUI window. VoiceOver cannot reach the web content, and hit-testing over it returns the AccessKit node underneath.

Setup: accesskit_macos 0.26.3, GPUI (gpui-kit 0.9.1, a snapshot of Zed's GPUI), which installs SubclassingAdapter on its NSView. gpui-webview adds the WKWebView as a subview of that view. macOS 27, Apple silicon.

What we tried

A local patch to subclass.rs (below):

  • children appends the visible native subviews after the AccessKit nodes. They go through NSAccessibilityUnignoredChildren, so a plain container view is skipped.
  • hit_test asks the native subview under the point first. It walks through ignored container views by hand: a container that lets mouse events through does not hit-test its own subviews.

Checked with an AXUIElement dump:

  • each body becomes an AXWebArea with its headings, paragraphs and links;
  • hit-testing over a body returns its web area;
  • over the list drawn through AccessKit, hit-testing still returns the row.

Limit

The native views sit beside the AccessKit nodes, not inside the node they belong to (the message that holds the body). In sequential navigation, VoiceOver reads every AccessKit node first, then the bodies. AccessKit has no node that stands for a native view.

Question

Would you take a PR in this shape? Or would you rather have a way for a node to stand for a native view, exposed in its place on macOS, so that embedded content follows the reading order?

The same local patch also maps Role::Heading to AXHeading; #830 already does that.

Patch against 0.26.3 
--- a/adapters/macos/src/subclass.rs
+++ b/adapters/macos/src/subclass.rs
@@ -18,8 +18,9 @@
     runtime::{AnyClass, Sel},
     sel,
 };
-use objc2_app_kit::{NSView, NSWindow};
-use objc2_foundation::{NSArray, NSObject, NSPoint};
+use objc2_app_kit::{NSAccessibilityUnignoredChildren, NSView, NSWindow};
+use objc2::msg_send;
+use objc2_foundation::{NSArray, NSObject, NSPoint, NSRect};
 use std::{cell::RefCell, ffi::c_void, sync::Mutex};
 
 use crate::{Adapter, event::QueuedEvents};
@@ -87,15 +88,67 @@
 unsafe extern "C" fn superclass(this: &NSView, _cmd: Sel) -> Option<&AnyClass> {
     let associated = associated_object(this);
     associated.ivars().prev_class.superclass()
+}
+
+// A native subview, such as a WKWebView an app embeds in its custom-drawn view,
+// is not in the AccessKit tree. It is exposed beside the tree's nodes, so
+// assistive technology can reach its own accessibility elements.
+fn native_subviews(view: &NSView) -> Vec<Id<NSView>> {
+    unsafe { view.subviews() }
+        .to_vec_retained()
+        .into_iter()
+        .filter(|subview| !unsafe { subview.isHidden() })
+        .collect()
+}
+
+/// The element under `point` (screen coordinates) inside the topmost native
+/// subview there. Ignored container views are walked through by hand: one that
+/// lets mouse events through to the view below would not hit-test its subviews.
+fn native_hit_test(view: &NSView, point: NSPoint) -> Option<*mut NSObject> {
+    for subview in native_subviews(view).into_iter().rev() {
+        let frame: NSRect = unsafe { msg_send![&subview, accessibilityFrame] };
+        let inside = point.x >= frame.origin.x
+            && point.x < frame.origin.x + frame.size.width
+            && point.y >= frame.origin.y
+            && point.y < frame.origin.y + frame.size.height;
+        if !inside {
+            continue;
+        }
+        let element: bool = unsafe { msg_send![&subview, isAccessibilityElement] };
+        if element {
+            return Some(unsafe { msg_send![&subview, accessibilityHitTest: point] });
+        }
+        if let Some(hit) = native_hit_test(&subview, point) {
+            return Some(hit);
+        }
+    }
+    None
 }
 
 unsafe extern "C" fn children(this: &NSView, _cmd: Sel) -> *mut NSArray<NSObject> {
     let associated = associated_object(this);
     let mut state = associated.ivars().state.borrow_mut();
     let state_mut = &mut *state;
-    state_mut
+    let nodes = state_mut
         .adapter
-        .view_children(&mut *state_mut.activation_handler)
+        .view_children(&mut *state_mut.activation_handler);
+    drop(state);
+    let subviews = native_subviews(this);
+    if subviews.is_empty() {
+        return nodes;
+    }
+    let mut all = unsafe { nodes.as_ref() }
+        .map(|nodes| nodes.to_vec_retained())
+        .unwrap_or_default();
+    // A plain container view is ignored by AppKit: expose what it holds instead.
+    let subviews: Id<NSArray<NSObject>> = NSArray::from_vec(
+        subviews.into_iter().map(|v| Id::into_super(Id::into_super(v))).collect(),
+    );
+    let unignored =
+        unsafe { NSAccessibilityUnignoredChildren(Id::cast::<NSArray>(subviews).as_ref()) };
+    let unignored: &NSArray<NSObject> = unsafe { unignored.cast().as_ref() };
+    all.extend(unignored.to_vec_retained());
+    Id::autorelease_return(NSArray::from_vec(all))
 }
 
 unsafe extern "C" fn focus(this: &NSView, _cmd: Sel) -> *mut NSObject {
@@ -106,6 +159,9 @@
 }
 
 unsafe extern "C" fn hit_test(this: &NSView, _cmd: Sel, point: NSPoint) -> *mut NSObject {
+    if let Some(hit) = native_hit_test(this, point) {
+        return hit;
+    }
     let associated = associated_object(this);
     let mut state = associated.ivars().state.borrow_mut();
     let state_mut = &mut *state;
主要语言
Rust
星标
1.6k
派生
135
平均合并
10 小时 53 分钟
30 天内合并 PR
11

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

AccessKit/accesskit 的其他 Issue

查看 AccessKit/accesskit 的全部 Issue

相似的 Issue

更多 Rust Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。