nvim ui thread freeze: 100% cpu rebuild_unique_words

Start with lua/cmp_buffer/buffer.lua at rebuild_unique_words and get_words, then inspect lua/cmp_buffer/source.lua where get_words is called. Reproduce the insert-mode CPU freeze and determine why the shown range iteration does not return; done means the freeze is explained and corrected without regressing buffer-word completion.

My nvim is rarely gets stuck when I enter insert mode with 100% cpu, I decided to debug it, and after a long session of lldb attaching to the nvim process and staring at assembly without symbols this is the stacktrace that I was able to extract from the relevant thread:

stack traceback:
        ...cal/share/nvim/lazy/cmp-buffer/lua/cmp_buffer/buffer.lua:357: in function 'rebuild_unique_words'
        ...cal/share/nvim/lazy/cmp-buffer/lua/cmp_buffer/buffer.lua:342: in function 'get_words'
        ...cal/share/nvim/lazy/cmp-buffer/lua/cmp_buffer/source.lua:66: in function ''
        vim/_editor.lua: in function ''
        vim/_editor.lua: in function <vim/_editor.lua:0>%

The code of the trace back:

function buffer.rebuild_unique_words(self, words_table, range_start, range_end)
  for i = range_start + 1, range_end do -- <<<<< seen also this line
    for _, w in ipairs(self.lines_words[i] or {}) do
      words_table[w] = true -- <<<<< this line
    end
  end
end

function buffer.get_words(self)
  -- NOTE: unique_words are rebuilt on-demand because it is common for the
  -- watcher callback to be fired VERY frequently, and a rebuild needs to go
  -- over ALL lines, not just the changed ones.
  if self.unique_words_other_lines_dirty then
    clear_table(self.unique_words_other_lines)
    self:rebuild_unique_words(self.unique_words_other_lines, 0, self.last_edit_first_line)
    self:rebuild_unique_words(self.unique_words_other_lines, self.last_edit_last_line, self.lines_count) -- << this line
    self.unique_words_other_lines_dirty = false
  end
  if self.unique_words_curr_line_dirty then
    clear_table(self.unique_words_curr_line)
    self:rebuild_unique_words(self.unique_words_curr_line, self.last_edit_first_line, self.last_edit_last_line)
    self.unique_words_curr_line_dirty = false
  end
  return { self.unique_words_other_lines, self.unique_words_curr_line }
end

vim.defer_fn(function()
    local input = string.sub(params.context.cursor_before_line, params.offset)
    local items = {}
    local words = {}
    for _, buf in ipairs(bufs) do
      for _, word_list in ipairs(buf:get_words()) do -- <<<< this line
        for word, _ in pairs(word_list) do
          if not words[word] and input ~= word then
            words[word] = true
            table.insert(items, {
              label = word,
              dup = 0,
            })
          end
        end
      end
    end

I can only imagine that range_start and range_end is a huge range that the loop just never exists, I'm not sure though.
It happens right when I enter insert mode and almost nonreproducible.

Edit: I am not somehow able to dump some values below, it seems like the last_edit_first_line and last_edit_last_line are very close, so might not be related, but the nvim always stops in this function and I never saw it leave so I'm not sure what's causing this, it's pretty hard to get information since I'm executing lua scripts from lldb with raw addresses in memory:

{
  bufnr = 7,
  closed = false,
  last_edit_first_line = 26,
  last_edit_last_line = 27,
  lines_count = 377,
  lines_words = { ... },
  on_close_cb = <function 1>,
  opts = {
    get_bufnrs = <function 2>,
    indexing_batch_size = 1000,
    indexing_interval = 100,
    keyword_length = 3,
    keyword_pattern =  "\\\\%(-\\\\?\\\\d\\\\+\\\\%(\\\\.\\\\d\\\\+\\\\)\\\\?\\\\|\\\\h\\\\%(\\\\w\\\\|á\\\\|Á\\\\|é\\\\|É\\\\|í\\\\|Í\\\\|ó\\\\|Ó\\\\|ú\\\\|Ú\\\\)*\\\\%(-\\\\%(\\\\w\\\\|á\\\\|Á\\\\|é\\\\|É\\\\|í\\\\|Í\\\\|ó\\\\|Ó\\\\|ú\\\\|Ú\\\\)*\\\\)*\\\\)\",
    max_indexed_line_length = 40960
  },
  regex = <userdata 1>,
  timer = {
    handle = <userdata 2>,
    <metatable> = {
      __index = {
        close = <function 3>,
        is_active = <function 4>,
        new = <function 5>,
        start = <function 6>,
        stop = <function 7>
      }
    }
  },
  timer_current_line = 368,
  unique_words_curr_line = {
    dap = true
  },
  unique_words_curr_line_dirty = true,
  unique_words_other_lines = {
    ...
  },
  unique_words_other_lines_dirty = true,
  words_distances = {},
  words_distances_dirty = true,
  words_distances_last_cursor_row = 0,
  <metatable> = {
    __index = {
      GET_LINES_CHUNK_SIZE = 1000,
      close = <function 8>,
      get_words = <function 9>,
      get_words_distances = <function 10>,
      index_line = <function 11>,
      index_range = <function 12>,
      mark_all_lines_dirty = <function 13>,
      new = <function 14>,
      rebuild_unique_words = <function 15>,
      safe_buf_call = <function 16>,
      start_indexing_timer = <function 17>,
      stop_indexing_timer = <function 18>,
      watch = <function 19>
    }
  }

}

Possibly a LuaJit bug, although I’m running latest, can’t be certain about how can this simple loop can lead to a LuaJit bug but I can’t find an explanation for the loop not exiting.