Dense Qwen retains duplicate MLP intermediates across serialized graph capacities

Đang mở
  • Có gì không ổn?
#45 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
35/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python
Lĩnh vực
machine-learning

Hướng nghiên cứu

Read the offered branch linked in the issue and compare its port with upstream commit 1fd1f91; the issue identifies the dense Qwen decoder and existing prefill/decode Scratch banks as the relevant entry points. Start with tests/recipes/test_qwen38_dense_scratch.py, then qualify the exact current-main tree with GPU preparation, capture, export, numerical checks, and native replay. Done means those checks pass; the issue says they are still outstanding.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

area:memory area:preparation model:qwen3.8-27b performance triage:verify-for-closure

Code offered for maintainer discussion: https://github.com/Maxrio9/flare-local-inference/tree/dense-mlp-scratch-20261023. This is an issue rather than a PR because the standalone current-main port has not received GPU qualification.

The dense Qwen decoder allocates gate, up, down and MLP-input intermediates inside every PreparedDecoder, retaining duplicate buffers across serialized graph capacities. Sharing only the CUDA graph pool does not share those constructor allocations. This contributes to preparation OOM on a 32GB card with the official Qwen3.8-27B target plus a parallel drafter at prefill 4096.

Allocate those four disjoint BF16 fields in each existing prefill/decode Scratch bank and pass bounded row views to graph variants. They share the same serialized lifetime as the bank's existing activations and GDN scratch; persistent model state is unchanged. The memory report includes retained mlp_scratch_bytes. No projection arithmetic, precision, graph contract or runtime change.

Offered implementation and validation status

The offered branch is a focused port of 67aec46 onto upstream 1fd1f91, without the MTP, DFlash2, attention-buffer or third-party target-loading changes. CPU storage validation checks distinct live fields, common backing across capacity views, contiguous rows and untouched tails.

Exact rebased tree: pytest -q tests/recipes/test_qwen38_dense_scratch.py passed (1 test), using the current pinned FlashInfer 9c9e94ef source for CPU imports in an isolated path. The shared experiment environment was left unchanged. The test does not initialize CUDA or qualify graph execution.

The original implementation on the experimental DFlash2 stack calculated a 1,688,207,360-byte (1.572 GiB) prefill-storage saving at chunk 4096, before optional scored variants. That is an allocation delta for that catalog, not a live-memory peak or a measured saving on this rebased ordinary catalog.

On that original stack: 69 state/ring CUDA cases, 21 MTP cases, compiled multi-chunk scoring and inactive-hole C4 checks passed. An official-target C4/64K/prefill4096 package exported and passed 222 pressure/probe requests, plus 75 on the final scheduler runtime and 51K cold/warm reuse. Cold 51K TTFT remained approximately 14.5–14.8 seconds: no latency improvement is claimed.

Not ready as a recipe PR: the exact standalone current-main port still needs GPU preparation/capture/export, alternating graph/scored-prefill numerical checks and native replay. Original-stack GPU results do not qualify this tree. Existing exported packages are unchanged; reduced buffers require re-export. Only dense Qwen is modified; other recipes' workspace lifetimes are not inferred from it.

Would you like this small ordinary-dense port qualified and submitted before the larger DFlash2 integration?

Authored and lifetime-reviewed by Codex (GPT-6), AI-assisted. Claude reviews measurement and wording only for this submission; the scratch design is not attributed to Claude Opus 5.5.

Ngôn ngữ chính
Rust
Star
800
Fork
106
Merge trung bình
1 ngày 3 giờ
Pull request đã merge (30 ngày)
100

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/hyperflare-inference

Tất cả issue của NVIDIA/hyperflare-inference

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.