ggml-org/llama.cpp

mpi : attempt inference of 65B LLaMA on a cluster of Raspberry Pis

Open

#2,164 opened on Jul 10, 2023

View on GitHub
 (54 comments) (29 reactions) (0 assignees)C++ (18,202 forks)batch import
hardwarehelp wantedresearch 🔬

Repository metrics

Stars
 (110,169 stars)
PR merge metrics
 (Avg merge 6d 8h) (389 merged PRs in 30d)

Description

Now that distributed inference is supported thanks to the work of @evanmiller in #2099 it would be fun to try to utilize it for something cool. One such idea is to connect a bunch of Raspberry Pis in a local network and run the inference using MPI:

# sample cluster of 8 devices (replace with actual IP addresses of the devices)
$ cat ./hostfile
192.168.0.1:1
192.168.0.2:1
192.168.0.3:1
192.168.0.4:1
192.168.0.5:1
192.168.0.6:1
192.168.0.7:1
192.168.0.8:1

# build with MPI support
$ make CC=mpicc CXX=mpicxx LLAMA_MPI=1 -j

# run distributed inference over 8 nodes
$ mpirun -hostfile ./hostfile -n 8 ./main -m /mnt/models/65B/ggml-model-q4_0.bin -p "I believe the meaning of life is" -n 64

Here we assume that the 65B model data is located on a network share in /mnt and that mmap works over a network share. Not sure if that is the case - if not, then it would be more difficult to perform this experiment.

Looking for people with access to the necessary hardware to perform this experiment

Contributor guide

Research direction
Set up a Raspberry Pi cluster with MPI support, ensure the model file is accessible via a network share with mmap support, build llama.cpp with LLAMA MPI=1, and run mpirun with the hostfile. Debug any issues related to MPI communication or mmap over network.
Tech stack
cppshell
Domain
backendinfrastructuredistributed systems
Issue type
Research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
5
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
Raspberry Pi cluster (8+ nodes)Network storage (NFS or similar) with mmap supportMPI installation (OpenMPI or MPICH)C/C++ build tools
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20