Feature Request: Better Braille Support

QEMU has the ability to pass braille output from a screen reader in a virtual machine to a braille display connected to the host by creating a fake USB braille device for the guest and sending its output to BRLAPI on the host; it supports sending braille key presses from BRLAPI to the guest screen reader as well. Two virtual devices are used for this: usb-braille, the virtual USB device, and a chardev called braille, which connects to BRLAPI. On Linux, BRLTTY is usually the program that connects to braille displays, and although it only supports the console directly, it allows other programs to take over the braille display with BRLAPI. For example, Orca, a screen reader for GUI programs, supports braille by connecting to BRLAPI, and so does Speechd-el, a screen reader for Emacs. When a client connects to BRLAPI, it sends the number of the virtual console it is on, so when you switch consoles the braille output switches as well. For instance, you could have a plain console on TTY1 with braille output from only BRLTTY, Emacs on TTY2 with braille output from Speechd-el, the Mate desktop running Firefox on TTY3 with braille output from Orca, and QEMU on TTY4 running Windows with braille output from NVDA. On MacOS, VoiceOver can connect to braille displays and enables braille input and output, but it does not have anything like BRLAPI to enable other programs to support braille directly. Also, UTM does not seem to support adding this QEMU braille device, since adding virtual USB devices is not mentioned in the manual, and I cannot find it in the VM settings or in the toolbar once you start a VM.

Although you can use USB sharing to connect a USB braille display to the guest, you would not be able to get braille output from VoiceOver until you give the device back to the host or stop the VM, nor could you run two virtual machines at once and use braille with both of them unless you disconnect the USB device from one and connect it to the other each time. This method also would not work with Bluetooth braille displays, and connecting a braille display using Bluetooth is very useful if you have a laptop and a braille display that you carry around with you or take different places often.

You can compile BRLTTY for MacOS and I think you can use it with both USB and Bluetooth braille displays. It supports the console if you also compile Screen with a special patch, as well as BRLAPI. If the guest operating system can run BRLTTY, you could connect to BRLAPI on the host through TCP, and NVDA on Windows supports BRLAPI as well, so you could either forward the remote port to localhost or run a BRLTTY instance on Windows that sends NVDA's braille to the remote port. Although this is better than my previous method for using braille with Windows and Linux virtual machines, it still has some significant shortcomings. First, there would be no way for BRLTTY on MacOS to know which window is focused, so if you run more than one VM or a VM and a console window with Screen, there would be no way to switch braille between them. Also, you would not be able to use braille with VoiceOver while the MacOS BRLTTY instance is running, although you can still use speech if you turn off VoiceOver before starting BRLTTY and turn it back on again. You could write a shortcut in AppleScript that switches between BRLTTY and VoiceOver braille, but you would probably have to restart BRLTTY or NVDA inside the VM every time.

To make what is possible on Linux with BRLTTY and QEMU possible on MacOS with VoiceOver and UTM, I think there are a few possibilities. The first step for any of these would probably be to add the ability to add the virtual braille device in UTM, maybe in the input tab, or a separate devices tab could be added that would allow people to add the USB braille device as well as other QEMU devices currently not supported by UTM, such as the Virtio memory balloon.

One possibility would be to make UTM display the output from the braille chardev through VoiceOver instead of sending it over a BRLAPI connection. I am not sure how this could be done, but there are IOS apps that have braille commands specific to the app, and some that show messages in braille that are not spoken, so I think there must be a way to make VoiceOver show a message in braille without speaking it on MacOS, and maybe accept braille input as well. This would allow one to get braille output from virtual machines while being able to use VoiceOver braille in other apps, with USB and Bluetooth braille displays, and one could run two VMs at the same time and use braille with both by switching windows.

Another possibility would be to send the braille output from the VM through BRLAPI like QEMU does, but add the ability for BRLTTY to know what window is in focus on MacOS and for UTM (and perhaps Screen as well if a native MacOS console driver is not written) to send its window number to BRLTTY. This would allow one to run two virtual machines and one or more MacOS terminal windows and get braille output from all of them, although not from VoiceOver. To get braille output from VoiceOver, either the ability to suspend itself and restart VoiceOver when neither a VM or console window is in focus, and to take the braille device back from VoiceOver (probably by restarting VoiceOver) when one of these windows gains focus to BRLTTY, or a braille driver for BRLTTY would have to be written that uses VoiceOver's braille support instead of communicating with a braille display directly, which I am not sure is possible.

What I am trying to do is make braille as usable as other input and output methods like the keyboard, the screen, and sound. I think my second possibility would be the best, but probably the hardest to implement. An example situation would be using a braille display with an M1 MacBook over Bluetooth, running some Mac GUI apps such as Safari or Pages, one or more Mac terminal windows, a Windows 10 virtual machine with NVDA running some GUI apps such as Microsoft Word and Outlook and one or more Windows consoles, and a Linux virtual machine with Orca and the Mate desktop on one TTY, Emacs with Speechd-el on another, and some text consoles with plain BRLTTY, all with good braille support that allows one to switch between them easily. With my second possibility, you could also use BRLTTY with the terminal on MacOS (with Screen or a native console driver) instead of VoiceOver braille since BRLTTY is better with terminals than VoiceOver, and you could also probably run the Android emulator and use BRLTTY in Android and connect to the MacOS BRLTY instance with BRLAPI. I know not all of this is related to UTM, but I am not sure where else to post this, and this would be very useful for blind users.

I do not have a Mac yet since I am waiting for the new MacBook Pro to come out, but I do have a 2015 MacBook that I am borrowing from a library for a few weeks, and currently I use a Raspberry Pi as my main computer. One of the main reasons I am getting an M1 Mac is because UTM lets you run Windows and Linux virtual machines at near-native performance, and the M1 CPU has such high performance that it is practical to run both on a laptop, as well as being able to run IOS apps natively and the Android emulator, so an M1 Mac can literally run programs from all five major operating systems and have integration with Apple services like iCloud and Messages, especially with UTM's shared directory support which would let you access your documents folder in iCloud from Windows and Linux.