Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Idea for Discussion: Device Memory Management Primitives

Đang mở
  • Có gì không ổn?
#780 6 bình luận 2 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
8/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Đình trệ
Công nghệ
javascript
Lĩnh vực
api, machine-learning

Hướng nghiên cứu

This is an open design discussion about memory primitives for MLGraph and MLBuffer in the WebNN spec, not a bounded change. Start from the spec's MLGraph and MLBuffer/MLTensor sections and the related caching discussion the issue links to. Done would mean a recorded design decision from the maintainers, not a code patch, so a newcomer should first raise the question on the thread.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

feature request

The idea is perhaps future-looking, but I'd like to bring it up for discussion.

Motivations

  • Reduce the GPU/NPU memory required for completing a use case (e.g. text2image).
  • Reduce the memory copy overhead of loading model weights (e.g. on disk) into GPU/NPU
  • Enable processing of models whose weights can't fit in main memory as a whole (e.g. pipelining)

Real World Examples

Case 1: Text to Image

Text to Image use cases generally involves multiple models, represented as a three stage pipeline (consisting of at least 3 models).

Take FLUX to generate an image of 1024x1024 for example:

Stage Model(s) Example Model Weight Size Model's Resident Memory on GPU (Approx.)
1. Text to Embedding Google T5 XXL and CLIP-L; fp8 7GB 8-12GB
2. Diffusion / Denoising FLUX UNet; fp8 12GB 16GB
3. Image decoding FLUX Variational AutoEncoder; fp8 200MB 16GB

In the ideal case, all three stages fits in GPU memory (totaling > 32GB). This exceeds the limit of every consumer GPU (except Apple M series with a large unified memory).

The only practical way to run FLUX is to "load, compute, unload" each model into GPU in sequence, at the cost of "reinitialize" each stage for each text2image inference.

This reduces the required GPU memory max(required_memory_per_stage) from sum(required_memory_per_stage), and requires the main memory to fit sum(size_of_model_weight).

Note:

  • Stable Diffusion has the same architecture (three stages), and uses pipelining on consumer GPUs.
  • Using a larger image size increases the resident memory proportionally for stage 2 and 3.
Case 2: Mixture of Experts (MoE) model

MoE models are self-explanatory. They used multiple small models to produce one inference result (e.g. the next token in LLMs).

Only one small model (say 1/8 size of the entire model) need to reside in GPU memory at a time. Each small model sequentially computes their result, then the results are merged into a single output token.

A high level pseudo code:

while output != END_OF_TEXT:
  outputs = []
  for small_model of small_models:
    small_model.load_to_gpu()
    out = small_model.predict_next_token()
    outputs.append(out)
    small_model.unload_from_gpu()
 output_token = select_from_outputs(outputs)
 emit_to_caller(output_token)

If the GPU has enough memory, all of the small models can reside in memory (load and unload becomes no-op). If not, the small models are repeatedly load to and unload from gpu (usually to/from main memory).

Some LLMs adopt an architecture where the number of activated params (model weight that has to reside in GPU memory) is much smaller than the number of total params (total size of model weight). I believe they functions similarly to MoE at inference time from memory usage standpoint.

Examples:

  • Mixtral 8x7b
  • DeepSeek v2 (236B total params, 21B activated param per token)
Case 3: Model Streaming

I observe this technique when playing with Ollama.

If the model weight is too big to fit into main memory, the model weight are streamed from disk during inference (e.g. the model will be read from disk N times for N predicted tokens).

It's slow (bottlenecked by disk throughput), but does allow large models inference to happen.

Current WebNN API

Case 1 and 2 are feasible but not efficient. Model pipelining and partitioning involves destroying and rebuilding the compute graph.

For every inference (e.g. generate one image from a text prompt, predict one token using MoE model), we need to:

  • Copy model weights from JavaScript ArrayBuffers into WebNN service process
  • And, call platform-specific API to build (with an optional, potentially expensive, fuze/optimize pass)

Case 3 is infeasible because the entire model weight needs to be copied to WebNN service process before we can build a graph. We can fallback to model partitioning, and convert the problem to case 2.

API / Spec Implication

The topic for discussion :)

Two primitives?

  1. Swap a built MLGraph between GPU/NPU memory and main memory (e.g. PyTorch model.to(device))
  2. Memory mapping from file-on-disk to main memory (POSIX mmap equivalent?)

Related spec / APIs I can think of:

  • MLBuffer / MLTensor
  • Fetch: Obtain a mmap-ed response?
  • File System Access: mmap a file on disk (from FileSystemFileHandle)?
  • WebGPU: Shader's compilation cache
  • @reillyeon mentioned to me about introducing a caching mechanism for MLGraph (e.g. save it for later use and avoid repeated graph compilation). Said mechanism might help here.
Ngôn ngữ chính
Bikeshed
Star
566
Fork
72
Merge trung bình
6 ngày 5 giờ
Pull request đã merge (30 ngày)
1

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của webmachinelearning/webnn

Tất cả issue của webmachinelearning/webnn

Issue tương tự

Thêm issue về Backend & API Design

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.