Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Lend part of a card from a running engine and get it back, without a park or a reload

Abierto
  • ¿Algo no está bien?
#189 23 comentarios 1 reacción 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@T-Crypt ya está trabajando en esto.

Desde el 23/10/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

Project home: https://github.com/TacoTakumi/vram-lend

My setup

  • Ubuntu 24.04, NVIDIA driver 610.43.02, CUDA 13.
  • 2x RTX 3090 (24 GB), 1x RTX 5060 Ti (16 GB) and 2x RTX 4060 Ti (16 GB). 96 GB of VRAM, 128 GB RAM.
  • llama-swap serves everything. Most models run on llama.cpp. GLM-5.3-Flash and DeepSeek V4.1 Flash run on Project Maya, Qwen 3.8 Flash Next runs on Strata.
  • Right now Maya runs GLM-5.3-Flash and DSV41 as three llama-swap entries, on 2, 3 or 5 cards.
  • ComfyUI runs under llama-swap on the 5060 Ti.
  • VoiceManager runs on a 4060 Ti.

What I want to do

When I want ComfyUI or a voice model, I switch Maya from the 5 card entry to the 2 or 3 card entry. The model reloads and I lose the session cache. When I am done I switch back and it reloads again.

Maya keeps the experts in system RAM and a cache of them on the cards. It can already give part of that cache away inside its own process. The vision encoder starts only when a picture arrives, in VRAM that the model lends from its expert cache. I want the same thing between processes. Maya drops part of its cache on one card and keeps serving, only slower. ComfyUI or the voice model uses that memory. When the borrower is done, Maya takes the memory back and fills its cache again.

So the engine does not stop. It lends N MiB on one named card and gets it back later. Park is all or nothing and needs-mib is fixed at the grant, so neither covers this. It is close to the fit and scenes item on your roadmap, but here the holder keeps running.

Questions

  1. Would you take a lease that its holder can shrink and grow while it serves? The flow I have in mind:
  • A borrower asks for N MiB on card X.
  • Berth asks the holder to give back N MiB on card X.
  • The holder answers when the memory is free. Berth checks the driver read, as it does for a release now.
  • Berth grants the borrower a sized lease.
  • When the borrower ends, Berth tells the holder it can take the memory back.
  1. Should this go in the park protocol? Something like this would work for Maya and Strata:
POST /berth/lend    {"card": "0000:05:00.0", "mib": 8192}  -> 200 when the memory is free, 409 if the engine cannot give that much
POST /berth/return  {"card": "0000:05:00.0"}               -> 200 when the engine holds it again
GET  /berth/state   -> {"state": "hot", "lent": {"0000:05:00.0": 8192}}

  1. My model holds five cards and lends from one of them. Will the v0.4 lease table let one lease cover several cards, with a size for each card?

  2. I will build this in my own forks of Berth, Maya and Strata. If the shape fits where Berth is going, would you take the Berth side back as a PR?

Thanks!

Lenguaje dominante
Go
Estrellas
226
Forks
20
Merge medio
43 min
PR fusionados (30 d)
192

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de T-Crypt/berth

Todos los issues de T-Crypt/berth

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.