Include in Readme how to Pass Custom Arguments to `llama_cpp.server` in Docker

Open Beginner friendly
#1,029 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
1/5
Estimated time
Under an hour
Newbie friendliness
75/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Stale
Tech stack
docker, python
Domain
cli, documentation

Research direction

Look at the Dockerfile and existing README sections about running the server. The issue shows the incorrect command; the solution is to document the correct way to pass arguments to the llama_cpp.server application, not to uvicorn. Check how the server's entry point is defined and update the README with a working example.

Written by the indexing model from the issue text.

Description

documentation question
Title:

Issue with Passing Custom Arguments to llama_cpp.server in Docker

Issue Description:

Hello abetlen,

I've been trying to use your Docker image ghcr.io/abetlen/llama-cpp-python:v0.2.24 for llama_cpp.server, and I encountered some difficulties when attempting to pass custom arguments (--n_gpu_layers 81, --chat_format chatml, --use_mlock False) to the server through Docker.

Steps to Reproduce:
  1. Pull the Docker image: docker pull ghcr.io/abetlen/llama-cpp-python:v0.2.24

  2. Run the container with custom arguments:

    docker run --rm -it -p 8000:8000 \
      -v /home/jaredquek/text-generation-webui/models:/models \
      -e MODEL=/models/tulu-2-dpo-70b.Q5_K_M.gguf \
      --entrypoint uvicorn \
      ghcr.io/abetlen/llama-cpp-python:v0.2.24 \
      --factory llama_cpp.server.app:create_app --host 0.0.0.0 --port 8000 --n_gpu_layers 81 --chat_format chatml --use_mlock False
    

    This results in an error: Error: No such option: --n_gpu_layers.

Expected Behavior:

I expected to be able to pass these arguments to the llama_cpp.server application inside the Docker container.

Actual Behavior:

The uvicorn command does not recognize these arguments as it's designed for the ASGI server, not the llama_cpp.server application.

Potential Solutions:
  • Modify the Dockerfile or application configuration to accept these arguments.
  • Provide guidance in Readme on how to correctly pass additional arguments or configure the server with these settings.

I would appreciate any assistance or guidance you could provide on this issue.

Thank you for your time and for maintaining this project.

Best regards.

Dominant language
Python
Stars
10.6k
Forks
1.5k
Avg merge
23m
Merged PRs (30d)
1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from abetlen/llama-cpp-python

All issues in abetlen/llama-cpp-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.