macOS on Apple silicon: `--gpus all` as a private channel to a GPU server on the Mac

Open
#356 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
macos, rust

Research direction

Read the issue’s linked patch against commit 260cf66 and the repository’s CONTRIBUTING guidance on cross-platform features. Start by tracing how GPU device requests are handled and how the guest agent and daemon communicate; the issue identifies nine changed files and added unit tests. Done means the maintainers agree on the macOS-only relay approach and its platform behavior, then the change meets the project’s support and validation requirements.

Written by the indexing model from the issue text.

Description

What you're trying to do

I run MLX and PyTorch programs in containers on an Apple-silicon Mac. In a container they compute on the CPU, because Metal does not reach into the Linux virtual machine. I wrote two bridges that send a program's array operations from the container to a process on the Mac, which computes them on the Apple GPU; the programs are not changed (https://github.com/FabioMalpezzi/varco).

With Docker Desktop the container reaches that process on a TCP port of the Mac, and so does every other container. With Kiri I wanted the container started with --gpus all, and only that one, to have the channel.

What you'd like Kiri to do

Accept a GPU device request on an Apple-silicon Mac, where Kiri refuses it today, and give that container a private channel to a Unix socket on the Mac. Kiri carries bytes and does nothing with the GPU itself.

I have this working as a patch against 260cf66 (main on 16 November): https://github.com/FabioMalpezzi/varco/tree/main/kiri

  • A container created with --gpus all (or Compose's capabilities: [gpu]) gets a listener at 127.0.0.1:47700 in its own network namespace, opened by the guest agent before the program starts, and the variable KIRI_GPU_BRIDGE.
  • Each connection is relayed over vsock to the daemon, which connects it to ~/.kiri/run/gpu.sock, where a server of the user's listens. With no server there the container sees its connection close.
  • --gpus together with --network host or --network container:... is refused, and no container may join the network namespace of a --gpus container, at create and at start.
  • On every other host a GPU request is refused as it is now.

It is nine files, with unit tests for the request and for the refusals. Through it, on an M4 Max, BGE-M3 embeddings took 1.01 to 1.02 times the Mac's time (1.11 through Docker Desktop's TCP port), and a 35B model generated the Mac's text at the Mac's speed. The figures and the method are in https://github.com/FabioMalpezzi/varco/blob/main/docs/benchmarks.md

What I know does not fit your rules as they stand, and would like your view on before any pull request:

  • It is for one system. On Windows and Linux hosts there is no Metal to reach, so the request stays refused there, and CONTRIBUTING asks that a feature ship on all three.
  • What Kiri would gain is a relay: a container asks for a device and gets a socket on the host. The names (KIRI_GPU_BRIDGE, port 47700, gpu.sock) are mine and can change.
  • I ran the unit tests the patch adds or changes, and the channel on a daemon built from source at 260cf66. I did not run tests/e2e.sh, the Speed check, or builds for Linux, Intel Macs and Windows.

I am not a native English speaker, so I used an AI tool to write this message. The patch was also written with the help of an AI coding tool; I built it, ran it and measured with it on my own Mac.

Would you like to build it?

Yes, once we've agreed on the approach

Dominant language
Rust
Stars
7.5k
Forks
329
Avg merge
7h 48m
Merged PRs (30d)
105

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from libracoder/kiri

All issues in libracoder/kiri

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.