Add n_keep parameter to LLama constructor to enable Streaming-LLM
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 65/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Stale
- Domain
- ai, ai-infra-agents, backend-api-design
Research direction
Look at the Llama class constructor in the high-level API to see where parameters are defined. The low-level API example shows how llama_load_model_from_file is called with lparams. Add n_keep to the constructor, pass it to the model loading parameters, and ensure it's documented. Test by creating a model with the keep parameter and verifying it's passed correctly.
Written by the indexing model from the issue text.
Description
A recent paper by Meta/MIT/CMU proposed StreamingLLM, a simple yet efficient solution to enable "infinite" context. Better yet, the implementation in llama.cpp is as trivial as changing the n_keep value with option --keep as discussed in this issue. Unfortunately, the high-level API of llama-cpp-python does not support the keep/n_keep parameter.
It should be simple to add the parameter to the high-level API, ideally in the constructor for class Llama and to pass it along to function llama_cpp.llama_load_model_from_file as part of parameter lparams here.
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 1.5k
- Avg merge
- 23m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from abetlen/llama-cpp-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2352 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2211 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2210 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2145 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
abetlen/llama-cpp-python#2135 · 4 reactions ·
All issues in abetlen/llama-cpp-python
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100