Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Proposal: API to Separate Graph Building from Weight Loading to Reduce Peak Memory Usage

Đang mở
  • Có gì không ổn?
#901 7 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
6/100
Loại issue
Tính năng
Độ rõ ràng
Cần làm rõ
Mức độ hoạt động
Đình trệ
Công nghệ
javascript
Lĩnh vực
ai, api, performance

Hướng nghiên cứu

This is a design proposal, not a bug to fix. Start with the proposed builder.constant(), builder.build(), graph.setConstantData() and context.dispatch() semantics, and check whether the backends can support weightless graph creation. The thread has been quiet for about 300 days with no linked pull request, so ask the maintainers whether the direction is still wanted before writing anything. Done means an agreed spec change, not a patch.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

feature request
Problem Statement

While analyzing the memory-consumption of the Stable Diffusion demo on a dGPU system, I observed that the CPU memory used during graph building is more than 3x the actual model size. Similar high-memory-usage behavior (requiring multiple times the model size) is also observed on iGPU systems, although the exact multiplier may differ.

While this might be irrelevant for smaller models, it presents a significant overhead for larger ones. For example:

  • A 4GB model can consume over 12GB of CPU memory during its load phase on a dGPU.
  • This is already critical on systems with 16GB of total memory.

This issue is even more pronounced on UVM (Unified Memory) systems where the CPU and GPU/NPU share memory. This high peak usage during loading unnecessarily consumes memory that could otherwise be allocated for the model weights and intermediate buffers on the device.

Proposal

I'd like to propose introducing an API that splits graph creation from weight passing (i.e., loading the constant data).

The primary goal is to enable streaming weights directly into the graph during initialization. This would avoid creating copies of the weights in JavaScript or in the backend's staging memory before the graph is fully built. Ideally, the weight data would only be copied once, from its source buffer directly to its final (e.g., device) location when the data is provided to the graph after it has been built.

This raises a key question for discussion:

  • Would all current WebNN backends be able to support a (mostly) "weightless" graph creation, where all tensor shapes and data types are known, but the actual weight data is not provided until a later step?
Potential Implementation Idea

One potential solution could look like this:

  1. Allow builder.constant() to be called with just an MLOperandDescriptor (defining the shape and data type), without requiring the ArrayBufferView data source.
  2. This call would return an MLOperand object that acts as a handle for this "hollow constant" (a tensor without its data).
  3. Allow builder.build() to succeed using MLOperands that represent these hollow constants, creating the executable MLGraph.
  4. Introduce a new method on the MLGraph interface, such as:
    graph.setConstantData(constantOperand, dataBuffer)
  5. After the MLGraph is built (step 3), the user must call this new setConstantData() method for every hollow constant they created. At this stage, the backend can perform any required conversions and transfer the dataBuffer directly to device memory.
  6. context.dispatch() would require all constants to be set. Attempting to call dispatch() before all hollow constants have been supplied with data (via setConstantData) would result in an error.
Expected Benefits

This change would significantly reduce peak memory pressure during initialization:

  • dGPU Systems: In an ideal scenario, this could limit the peak CPU memory overhead to 1x-2x the size of the largest single tensor (for temporary buffering during upload), rather than 3x the entire model.
  • iGPU/UVM Systems: The hope is that no temporary CPU-side storage would be needed for the "upload" (as it's shared memory). This would reduce the total peak CPU memory consumption down to roughly Model Size + Max Single Tensor Size.
Ngôn ngữ chính
Bikeshed
Star
566
Fork
72
Merge trung bình
6 ngày 5 giờ
Pull request đã merge (30 ngày)
1

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của webmachinelearning/webnn

Tất cả issue của webmachinelearning/webnn

Issue tương tự

Thêm issue về AI Infra & Agents

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.