`Matcher.highlight` highlights the wrong characters with markup or `İ`, and can raise `IndexError`

Open
#6,774 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
Half a day
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
cli

Research direction

Start in src/textual/fuzzy.py, in Matcher.highlight, where the candidate is lowercased and offsets are used to index the original text, and where the markup is parsed but the query is matched against the raw string. Run the two reproductions from the issue to see the wrong span and the IndexError. Done when offsets map back to the displayed text for both the İ case and the markup case, and the palette shows both commands for 'ir'.

Written by the indexing model from the issue text.

Description

Hi! I hit this while writing generated tests against Textual's widgets.

Matcher.highlight() styles offsets that don't index the text it displays, in two ways.

1. Characters that lowercase to two code points. "İ".lower() is "i̇" (i plus U+0307), so in a case-insensitive match the offsets index the lowered candidate, which is one longer than the original. The highlight lands one character late, and on the last character it raises IndexError:

from textual.fuzzy import Matcher

print(Matcher("z").highlight("İzmir").spans)
print(Matcher("r").highlight("İzmir"))

Output (the same on 8.2.7 and on main at 5e5b7ef):

[Span(2, 3, style=Style(reverse=True))]
Traceback (most recent call last):
  File "repro.py", line 4, in <module>
    print(Matcher("r").highlight("İzmir"))
  File ".../textual/fuzzy.py", line 217, in highlight
    if not candidate[offset].isspace():
IndexError: string index out of range

The first line highlights m rather than z.

In the command palette the IndexError is raised inside the provider's search task, so it isn't reported; the provider's remaining commands just disappear. With these two commands, typing ir shows neither:

import asyncio
from textual.app import App, SystemCommand
from textual.command import CommandList

class PaletteApp(App):
    def get_system_commands(self, screen):
        yield SystemCommand("Weather in İzmir", "Turkey", self.bell)
        yield SystemCommand("Weather in Izmir", "Turkey, ASCII spelling", self.bell)

async def main() -> None:
    app = PaletteApp()
    async with app.run_test() as pilot:
        await pilot.press("ctrl+p")
        await pilot.pause()
        await pilot.press("i", "r")
        await pilot.pause(0.5)
        command_list = app.screen.query_one(CommandList)
        print("commands shown for 'ir':", [str(option.prompt).splitlines()[0] for option in command_list.options])

asyncio.run(main())
commands shown for 'ir': []

With the fix below it prints ['Weather in İzmir', 'Weather in Izmir'].

2. Candidates containing markup. highlight() parses the candidate as markup, but matches the query against the raw string, so the offsets include the tags:

from textual.fuzzy import Matcher

content = Matcher("old").highlight("[b]Bold[/b] text")
print(repr(content.plain))
for span in content.spans:
    print(span, repr(content.plain[span.start:span.end]))

Output (the same on 8.2.7 and on main):

'Bold text'
Span(0, 4, style='b') 'Bold'
Span(4, 5, style=Style(reverse=True)) ' '
Span(5, 6, style=Style(reverse=True)) 't'
Span(6, 7, style=Style(reverse=True)) 'e'

I'd expect o, l and d of "Bold" to be highlighted.

Textual 8.2.7 and main (5e5b7ef), Python 3.13, macOS.

Dominant language
Python
Stars
37.7k
Forks
1.5k
PR merge metrics
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Textualize/textual

All issues in Textualize/textual

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.