LlamaPromptLookupDecoding gives no performance improvement

Open
#2,110 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
ai, performance

Research direction

Look at the llama-cpp-python bindings for the draft_model parameter and the LlamaPromptLookupDecoding class. Review the underlying llama.cpp implementation of prompt lookup decoding to understand its expected performance gain. Test with different prompt structures and model configurations to verify the feature is active and measure token generation speed.

Written by the indexing model from the issue text.

Description

I'm automating spelling correction. Currently I'm testing this on a Mistral 7B Instruction Q4_K_M model running in llama.cpp using python bindings, and it works fine. I submit text fragments up to ~2000 characters.

I bumped into the "draft_model" option that does "prompt lookup decoding" (a.k.a. token prediction). Because my output is very similar to my prompt text, I'd assume that activating this would bring me significant performance gain.

But I've been testing this out and I see no performance gain at all, comparing the inference with and without the 'draft_model' option activated. I've tried different values for 'num_pred_tokens', but that doesn't make any difference either.

Using llama.ccp 0.3.16. I'm running on CPU. My prompts consists of an instruction in English and then the text to be corrected in Dutch.

Maybe I'm doing something wrong? Or maybe I don't understand the purpose of this option?

Initialization:

    n_ctx = 4096
    
    llm = Llama(
        model_path=model_path,
        n_ctx=n_ctx,
        n_threads=8,
        temperature=0,
        chat_format=None,
        draft_model=LlamaPromptLookupDecoding(num_pred_tokens=3),
        logits_all=True,
    )

Inference:

    output = llm(
        prompt=prompt_text,
        max_tokens=2048,
        temperature=0,
        echo=True
    )
Dominant language
Python
Stars
10.7k
Forks
1.5k
Avg merge
1h 40m
Merged PRs (30d)
5

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from abetlen/llama-cpp-python

All issues in abetlen/llama-cpp-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.