Smaller docker image CUDA example
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 70/100
- Issue type
- Documentation
- Clarity
- Clearly specified
- Activity status
- Stale
- Tech stack
- docker, python
- Domain
- build-system, devops
Research direction
Look at the existing Dockerfiles in the repository, likely in a docker/ directory or referenced in the README. Compare the provided multi-stage Dockerfile to the current approach. Test building the image to verify the size reduction and that GPU support works. The goal is to integrate this improved Dockerfile into the project's documentation or examples.
Written by the indexing model from the issue text.
Description
ive been struggling some with a Cuda dockerfile sinze the devel image was so large the build ended up at almost 8gb, i came up with this. just wanted to share it:
FROM nvidia/cuda:12.1.1-devel-ubuntu22.04 AS builder
RUN apt-get update && apt-get upgrade -y \
&& apt-get install -y git build-essential \
python3 python3-pip gcc wget \
ocl-icd-opencl-dev opencl-headers clinfo \
libclblast-dev libopenblas-dev \
&& mkdir -p /etc/OpenCL/vendors && echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd
# setting build related env vars
ENV CUDA_DOCKER_ARCH=all
ENV LLAMA_CUBLAS=1
# Install depencencies
RUN python3 -m pip install --user --upgrade pip pytest cmake scikit-build setuptools fastapi uvicorn sse-starlette pydantic-settings starlette-context
# Install llama-cpp-python (build with cuda)
RUN CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install --user llama-cpp-python
FROM nvidia/cuda:12.1.1-runtime-ubuntu22.04
RUN apt-get update && apt-get upgrade -y \
&& apt-get install -y git build-essential \
python3 python3-pip gcc wget \
ocl-icd-opencl-dev opencl-headers clinfo \
libclblast-dev libopenblas-dev \
&& mkdir -p /etc/OpenCL/vendors && echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd
# Copy libraries from build local path
COPY --from=builder /root/.local /root/.local
COPY --from=builder /tmp/*.txt /tmp/
ENV PATH=/root/.local:$PATH
the final image ended up att aroung 2gb and GPU support is working as expected..
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 1.5k
- Avg merge
- 23m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from abetlen/llama-cpp-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2352 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2211 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2210 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2145 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
abetlen/llama-cpp-python#2135 · 4 reactions ·
All issues in abetlen/llama-cpp-python
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100