macOS on Apple silicon: `--gpus all` as a private channel to a GPU server on the Mac

Aperta
#356 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
macos, rust

Direzione di ricerca

Read the issue’s linked patch against commit 260cf66 and the repository’s CONTRIBUTING guidance on cross-platform features. Start by tracing how GPU device requests are handled and how the guest agent and daemon communicate; the issue identifies nine changed files and added unit tests. Done means the maintainers agree on the macOS-only relay approach and its platform behavior, then the change meets the project’s support and validation requirements.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

What you're trying to do

I run MLX and PyTorch programs in containers on an Apple-silicon Mac. In a container they compute on the CPU, because Metal does not reach into the Linux virtual machine. I wrote two bridges that send a program's array operations from the container to a process on the Mac, which computes them on the Apple GPU; the programs are not changed (https://github.com/FabioMalpezzi/varco).

With Docker Desktop the container reaches that process on a TCP port of the Mac, and so does every other container. With Kiri I wanted the container started with --gpus all, and only that one, to have the channel.

What you'd like Kiri to do

Accept a GPU device request on an Apple-silicon Mac, where Kiri refuses it today, and give that container a private channel to a Unix socket on the Mac. Kiri carries bytes and does nothing with the GPU itself.

I have this working as a patch against 260cf66 (main on 16 November): https://github.com/FabioMalpezzi/varco/tree/main/kiri

  • A container created with --gpus all (or Compose's capabilities: [gpu]) gets a listener at 127.0.0.1:47700 in its own network namespace, opened by the guest agent before the program starts, and the variable KIRI_GPU_BRIDGE.
  • Each connection is relayed over vsock to the daemon, which connects it to ~/.kiri/run/gpu.sock, where a server of the user's listens. With no server there the container sees its connection close.
  • --gpus together with --network host or --network container:... is refused, and no container may join the network namespace of a --gpus container, at create and at start.
  • On every other host a GPU request is refused as it is now.

It is nine files, with unit tests for the request and for the refusals. Through it, on an M4 Max, BGE-M3 embeddings took 1.01 to 1.02 times the Mac's time (1.11 through Docker Desktop's TCP port), and a 35B model generated the Mac's text at the Mac's speed. The figures and the method are in https://github.com/FabioMalpezzi/varco/blob/main/docs/benchmarks.md

What I know does not fit your rules as they stand, and would like your view on before any pull request:

  • It is for one system. On Windows and Linux hosts there is no Metal to reach, so the request stays refused there, and CONTRIBUTING asks that a feature ship on all three.
  • What Kiri would gain is a relay: a container asks for a device and gets a socket on the host. The names (KIRI_GPU_BRIDGE, port 47700, gpu.sock) are mine and can change.
  • I ran the unit tests the patch adds or changes, and the channel on a daemon built from source at 260cf66. I did not run tests/e2e.sh, the Speed check, or builds for Linux, Intel Macs and Windows.

I am not a native English speaker, so I used an AI tool to write this message. The patch was also written with the help of an AI coding tool; I built it, ran it and measured with it on my own Mac.

Would you like to build it?

Yes, once we've agreed on the approach

Lingua principale
Rust
Stelle
7.5k
Fork
329
Merge medio
7h 48m
PR unite (30g)
105

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di libracoder/kiri

Tutte le issue di libracoder/kiri

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.