RFC: Integrating KVMem virtual working set for 256K context on 16GB VRAM (Working prototype ready in fork)

未關閉
#192 27 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

評估

難度
5/5
預估耗時
一週以上
新手友好度
8/100
Issue 類型
功能
描述清晰度
基本清楚
活躍度
活躍
技術堆疊
cpp

研究方向

This is an RFC, not a scoped task: it asks the maintainer to choose between upstreaming a ~36K LOC KVMem working-set engine and keeping it as a downstream fork, and the 27-comment thread is where that decision is being made. To follow it, read the fork branch beellama-kvmem and the referenced ggml-cuda/gated_delta_net.cu Blackwell fix, then the thread for the maintainer's answer. Done means an architectural decision plus, at most, a targeted PR such as the GDN CUDA fix; nothing here is a first contribution.

由索引模型根據 Issue 內容生成。

描述

RFC: Integrating KVMem virtual working set for 256K context on 16GB VRAM (Working prototype ready in fork)

Context & Motivation

Large context windows (128K–256K) on 27B+ models like Qwen3.8-27B (and hybrid architectures with Gated Delta Net) are traditionally difficult to run on consumer hardware (16GB VRAM cards like RTX 4080 / RTX 5060 Ti).

While BeeLlama's native KVarN (and SWA) provides substantial cache savings (~2.67x with kvarn6), pushing beyond 100K–128K on 16GB VRAM still pushes right up against physical memory limits.

We have successfully integrated the KVMem virtual working set engine (arXiv:2609.04852) into BeeLlama v0.4.6, maintaining full compatibility with BeeLlama's core optimizations while allowing true 256K context streaming.

A fully working implementation, benchmarks, and Windows binaries are available in the fork:
👉 Repository & Branch: Andgihat/beellama.cpp (branch: beellama-kvmem)
👉 Release & Binaries: v0.4.6-kvmem


How it works
  1. Host RAM Virtual Working Set: The complete 262,144 (256K) context lives in pinned Host RAM (DDR5, ~8–12 GB).
  2. Fixed GPU Slot Pool: Only an active working set (e.g. Budget $B = 36K$ + Reserve $R = 16K$) sits in VRAM (~14.6 GB total with weights, zero OOM).
  3. Graph Recall & Cache Quantization: Blocks (128 tokens) are staged in/out over PCIe on the fly using Hadamard-rotated quantized slots (q4_0 / q8_0).
  4. Blackwell sm_120 Fix: In the process, we resolved a GDN warp reduction bug in ggml-cuda/gated_delta_net.cu (lanes_per_col stride and pointer offset check keep_rs = K > 1), restoring 100% correct mathematics for hybrid Qwen3.8 on RTX 50-series GPUs.

Benchmark Results (Qwen3.8-27B-UD-Q3_K_XL on RTX 5060 Ti 16GB)

Tested on real text documents from 10K to 256K context:

  • Prompt Processing (Prefill @ 10K+ tokens): 915.84 tokens/s (10,196 tokens processed in 11.13s with -b 2048 -ub 1024).
  • Decode Speed: 35.6 - 40.3 tokens/s with built-in MTP (--spec-type draft-mtp).
  • MTP Acceptance Rate: 76.0% (consistent 2.46 tokens/round).
  • RAM Usage @ 10K: Only 187 MB with q4_0 cache.
  • VRAM Headroom: Fits comfortably within 15.3 GB on a 16GB GPU.

Question for Maintainer (@Anbeeld)

Because the KVMem integration is relatively substantial (~36K LOC including tests, standalone llama-kvmem-server, and runtime adapters), we wanted to check with you first:

  1. Upstream PR: Would you be interested in having KVMem integrated directly into upstream BeeLlama (e.g. as an optional feature / server target)?
  2. Specialized Fork: Or would you prefer we maintain this as a specialized downstream fork / companion project (beellama-kvmem), and only submit targeted PRs (such as the Blackwell GDN CUDA fixes in gated_delta_net.cu) to upstream?

Happy to adapt the changes or submit targeted PRs depending on your architectural preferences. Thank you for the awesome work on BeeLlama!

主要語言
C++
星號
1.2k
分支
80
平均合併
4 天 20 小時
30 天內合併 PR
2

環境準備

這個專案沒有提供開發容器、Dockerfile 或貢獻指南,環境需要你自己搭建:先看它的 README,通用步驟見我們的新手貢獻指南。

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

Anbeeld/beellama.cpp 的其他 Issue

查看 Anbeeld/beellama.cpp 的全部 Issue

相似的 Issue

更多 C++ Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。