AT-SPI: a combo box's value, and expanded/collapsed states, don't reach screen readers

Start with the reproduction in the issue: run it under accesskit_atspi_common to see that the combo box's Node::set_value text and the set_expanded state never reach AT-SPI. Choosing between exposing the value through the Text interface or a selected child is a design decision the maintainers need to make first, and #665 already covers Expandable/Expanded/Collapsed. Done means the reproduction reports the value and the expanded/collapsed states, and the expand and collapse actions are offered.

🤖 Maddimax (AI)

Versions: accesskit 0.25.1, accesskit_atspi_common 0.21.0 (the adapter accesskit_unix 0.24 and accesskit_winit 0.34 use).

Two things a tree provides never reach AT-SPI clients such as Orca. macOS and UI Automation both expose them already.

1. A combo box's value

A collapsed combo box gets its current text with Node::set_value. On AT-SPI the value has nowhere to go:

• the object's name is its label, and only Role::Label takes its name from the value (label_comes_from_value);
• the Value interface is numeric only (current_value);
• the Text interface is only for nodes with text runs.

So Orca reads "Fruit, combo box" where VoiceOver and Narrator say "Fruit, combo box, Apple". UI Automation reads the value through the Value pattern, and macOS through accessibilityValue.

Orca reads a combo box's value from its selected child, or from its Text interface. Either of these would fix it:

• give a combo box with a value the Text interface, holding that value;
• or expose its selection: an object for the value, as GTK structures its combo boxes.

As a workaround we now give collapsed combo boxes a selected MenuListOption child on Linux only. We'd rather drop it.

2. Expandable, expanded and collapsed

Node::set_expanded maps to no AT-SPI state, so Orca can't tell whether a combo box's popup is open, or whether a tree item is expanded. #665 adds Expandable with Expanded or Collapsed, and with it the state-change events through notify_state_changes. This issue is mostly here to say it is wanted, and to link the combo box case above.

Related: Action::Expand and Action::Collapse aren't offered through the Action interface either. Only click is listed, below.

Reproduction

Cargo.toml:

[dependencies]
accesskit = "0.25"
accesskit_atspi_common = "0.21"

src/main.rs:

use accesskit::{
    Action, ActionHandler, ActionRequest, Node, NodeId, Rect, Role, TreeId, TreeInfo, TreeUpdate,
};
use accesskit_atspi_common::{
    Adapter, AdapterCallback, AppContext, Event, FullNodeId, InterfaceSet, PlatformNode, State,
    WindowBounds,
};

struct NoCallback;
impl AdapterCallback for NoCallback {
    fn register_interfaces(&self, _: &Adapter, _: FullNodeId, _: InterfaceSet) {}
    fn unregister_interfaces(&self, _: &Adapter, _: FullNodeId, _: InterfaceSet) {}
    fn emit_event(&self, _: &Adapter, _: Event) {}
}

struct NoActions;
impl ActionHandler for NoActions {
    fn do_action(&mut self, _: ActionRequest) {}
}

const WINDOW: NodeId = NodeId(0);
const COMBO_BOX: NodeId = NodeId(1);
const TREE_ITEM: NodeId = NodeId(2);

fn main() {
    let mut window = Node::new(Role::Window);
    window.set_children(vec![COMBO_BOX, TREE_ITEM]);

    // A collapsed combo box showing "Apple".
    let mut combo_box = Node::new(Role::ComboBox);
    combo_box.set_bounds(Rect::new(0.0, 0.0, 100.0, 30.0));
    combo_box.set_label("Fruit");
    combo_box.set_value("Apple");
    combo_box.set_expanded(false);
    combo_box.add_action(Action::Click);
    combo_box.add_action(Action::Expand);

    // An expanded tree item.
    let mut tree_item = Node::new(Role::TreeItem);
    tree_item.set_bounds(Rect::new(0.0, 40.0, 100.0, 70.0));
    tree_item.set_label("Documents");
    tree_item.set_expanded(true);
    tree_item.add_action(Action::Collapse);

    let update = TreeUpdate {
        nodes: vec![(WINDOW, window), (COMBO_BOX, combo_box), (TREE_ITEM, tree_item)],
        tree: Some(TreeInfo::new(WINDOW)),
        tree_id: TreeId::ROOT,
        focus: COMBO_BOX,
    };
    let adapter = Adapter::new(
        &AppContext::new(Some("repro".into())),
        NoCallback,
        update,
        true,
        WindowBounds::default(),
        NoActions,
    );
    let root = adapter.platform_node(adapter.root_id());
    let child = |index| root.relative(root.child_at_index(index).unwrap().unwrap());
    for node in [child(0), child(1)] {
        report(&node);
    }
}

fn report(node: &PlatformNode) {
    let state = node.state();
    println!("{:?} {:?}", node.role().unwrap(), node.name().unwrap());
    for flag in [State::Expandable, State::Expanded, State::Collapsed] {
        println!("  {flag:?}: {}", state.contains(flag));
    }
    println!("  interfaces: {:?}", node.interfaces().unwrap());
    println!("  children: {}", node.child_count().unwrap());
    let actions: Vec<String> = (0..node.n_actions().unwrap_or(0))
        .map(|i| node.action_name(i).unwrap_or_default())
        .collect();
    println!("  actions: {actions:?}");
}

Output:

ComboBox "Fruit"
  Expandable: false
  Expanded: false
  Collapsed: false
  interfaces: InterfaceSet(BitFlags<Interface>(0b100000000100011, Accessible | Action | Component | Selection))
  children: 0
  actions: ["click"]
TreeItem "Documents"
  Expandable: false
  Expanded: false
  Collapsed: false
  interfaces: InterfaceSet(BitFlags<Interface>(0b100001, Accessible | Component))
  children: 0
  actions: []

"Apple" can't be read through any interface. Neither node has Expandable, Expanded or Collapsed. Neither offers expand or collapse as an action.

Expected:

• the combo box's value can be read: as Text, or as a selected child;
• the combo box is Expandable and Collapsed;
• the tree item is Expandable and Expanded;
• the expand and collapse actions are offered where the node supports them.