Help wanted: Bi-directional text and Arabic shaping support

Our text rendering strategy cuts corners to achieve maximal performance in a web-based environment. We render glyphs to a texture atlas and then composite them on the GPU with a texture-mapped quad representing each character on screen. We currently composite characters with a naive strategy, just laying them out one after another from left to right.

Bi-directional text

Xray is a code editor, and most programming languages assume a left-to-right layout. We're not interested in supporting authoring an entire document with right-to-left lines. For to support programmers working in right-to-left locales, we do want the ability to embed strings of right-to-left text within left-to-right text, such as within strings and comments. This is called bi-directional text.

Interestingly, text in right-to-left languages is still stored from left-to-right in strings. There's an official algorithm defined by the Unicode standards body for transforming fragments of a string from this left-to-right "logical order" to the order the string should be displayed on screen. Anyone that wants to take on this issue will probably need to wrap their heads around the basics of how it works.

Arabic shaping

When we tested using HarfBuzz to perform a general text-shaping pass on every line, we ran into a couple problems:

• HarfBuzz is written in C, and it's not currently possible to compile a single WebAssembly module that contains both Rust and C/C++. At least we couldn't figure it out. Let us know if we're wrong.
• When we compiled HarfBuzz to its own module and tested running strings through it, it was taking between 4ms and 20ms to process 50 lines of 100 characters each, depending on the font.

Even if we solve these issues, there's still the issue of rendering glyphs. We currently render glyphs via an HTML5 canvas. In Electron, this simplifies a lot of cross-platform headaches related to loading fonts, dealing with fallbacks, etc. For the WebAssembly case, this is our only option unless we want to render everything via FreeType, which seems extremely complex.

However, canvas isn't actually capable of rendering glyphs. It can only render strings. So even if HarfBuzz or a platform-specific text shaping library returns a list of glyphs to render, we can't actually render an arbitrary glyph with canvas. Some glyphs in some languages are only accessible via context-specific combinations of letters.

Long story short, skipping generalized text shaping saves us a lot of complexity and performance overhead in the common case of editing code. What does it lose us?

• Support for rendering Arabic scripts
• Support for rendering Indic scripts

Both of these languages rely on contextual alternates. For example, Arabic characters can correspond to one of four different glyphs depending on their position within a word. Indic scripts combine syllables into individual glyphs in ways I don't really understand.

Indic script support is out of scope for this issue, but it turns out we can support Arabic without doing full-blown text shaping and font rasterization. Unicode happens to define "presentational" code points, which each represent a single contextual variant of a character. This great article from the Mapbox GL JS team describes how a normal string containing Arabic can be transformed to use these presentational forms, which we can then rasterize with our naive character strategy without issues.

What a solution might look like

Assuming we can't figure out how to compile C and Rust in the same WebAssembly module, we'd like a pure Rust solution that takes care of the bi-directional text transformation and the substitution of Arabic presentational characters in the output string.

There seem to be several implementations of "arabic shaping" floating around on the internet that could serve as inspiration, though a lot of them are GPL-licensed, so be careful about derivative works since Xray is MIT-licensed.

An additional challenge is that it won't be sufficient to simply transform the text. We'll also need to include enough metadata to understand where the cursor should be rendered inside a line containing bi-directional text and Arabic substitutions. I haven't thought deeply about how this mapping would be structured, but we essentially need to efficiently translate back and forth between columns in the input text and the output text.

A great solution would also contribute some kind of documentation explaining what you learn researching this problem.