Include in Readme how to Pass Custom Arguments to `llama_cpp.server` in Docker
Nobody has claimed this yet.
Assessment
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Newbie friendliness
- 75/100
- Issue type
- Documentation
- Clarity
- Clearly specified
- Activity status
- Stale
- Tech stack
- docker, python
- Domain
- cli, documentation
Research direction
Look at the Dockerfile and existing README sections about running the server. The issue shows the incorrect command; the solution is to document the correct way to pass arguments to the llama_cpp.server application, not to uvicorn. Check how the server's entry point is defined and update the README with a working example.
Written by the indexing model from the issue text.
Description
Title:
Issue with Passing Custom Arguments to llama_cpp.server in Docker
Issue Description:
Hello abetlen,
I've been trying to use your Docker image ghcr.io/abetlen/llama-cpp-python:v0.2.24 for llama_cpp.server, and I encountered some difficulties when attempting to pass custom arguments (--n_gpu_layers 81, --chat_format chatml, --use_mlock False) to the server through Docker.
Steps to Reproduce:
-
Pull the Docker image:
docker pull ghcr.io/abetlen/llama-cpp-python:v0.2.24 -
Run the container with custom arguments:
docker run --rm -it -p 8000:8000 \ -v /home/jaredquek/text-generation-webui/models:/models \ -e MODEL=/models/tulu-2-dpo-70b.Q5_K_M.gguf \ --entrypoint uvicorn \ ghcr.io/abetlen/llama-cpp-python:v0.2.24 \ --factory llama_cpp.server.app:create_app --host 0.0.0.0 --port 8000 --n_gpu_layers 81 --chat_format chatml --use_mlock FalseThis results in an error:
Error: No such option: --n_gpu_layers.
Expected Behavior:
I expected to be able to pass these arguments to the llama_cpp.server application inside the Docker container.
Actual Behavior:
The uvicorn command does not recognize these arguments as it's designed for the ASGI server, not the llama_cpp.server application.
Potential Solutions:
- Modify the Dockerfile or application configuration to accept these arguments.
- Provide guidance in Readme on how to correctly pass additional arguments or configure the server with these settings.
I would appreciate any assistance or guidance you could provide on this issue.
Thank you for your time and for maintaining this project.
Best regards.
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 1.5k
- Avg merge
- 23m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from abetlen/llama-cpp-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2352 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2211 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2210 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2145 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
abetlen/llama-cpp-python#2135 · 4 reactions ·
All issues in abetlen/llama-cpp-python
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100