[BUG] NPU grouped expert LoRA parameters lack expert-parallel allreduce metadata

Đang mở
  • Có gì không ổn?
#229 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
42/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python, pytorch
Lĩnh vực
distributed-systems, machine-learning

Hướng nghiên cứu

Start in src/mcore_bridge/tuners/npu_lora.py, in the NpuGroupedLoraLinear constructor where weight{i} and bias{i} are registered without an allreduce attribute, and check how is_expert is passed from src/mcore_bridge/tuners/lora.py. Run the constructor snippet from the issue to confirm the attribute is missing; this needs a Megatron/MindSpeed environment to be meaningful. Done means expert adapter parameters report allreduce=False while ordinary adapters are unchanged, and a test covers that classification.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug
Checklist / 检查清单
  • I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。
Bug Description / Bug 描述
Summary

The NPU grouped LoRA path appears to create per-expert adapter parameters without the parameter-level metadata Megatron uses to select the expert topology.

NpuGroupedLoraLinear stores self.is_expert, but registers weight{i} and optional bias{i} as plain nn.Parameter objects without setting param.allreduce. Megatron consumers fall back to getattr(param, "allreduce", True), so these parameters can be classified as ordinary dense parameters even when the module represents independent local experts.

Validation scope: this report is based on a source audit. I have not reproduced an NPU training failure or a model-quality regression. The constructor inspection below is provided for reproduction in an installed environment; it has not been executed here. An external MindSpeed patch could affect the final runtime behavior.

Source evidence
  • Constructor at commit 86a6a5b: module-level is_expert is retained, but the newly registered parameters receive no allreduce attribute. The current default-branch constructor has the same behavior.
  • LoRA construction path: the NPU grouped branch constructs both LoRA A and B with NpuGroupedLoraLinear(..., is_expert=self.is_expert).
  • Expert checkpoint mapping assigns separate global expert indices, so these adapters are per-expert parameters.
  • I could not find a subsequent adapter-wide setter for this attribute in the audited mcore-bridge path.

Relevant Megatron consumers:

Expected behavior and potential impact

For this is_expert=True, EP=2, ETP=1 case, adapter parameters should receive the expert-topology metadata, including allreduce=False, consistently with Megatron's expert layers. This still synchronizes replicas of the same expert through expert DP.

If nothing sets the metadata later, ordinary DP selection can synchronize parameters belonging to different experts when the two topologies differ. For example, with world_size=8, TP=2, EP=4, ETP=1, PP=1, CP=1 and rank order tp-cp-ep-dp-pp, the first ordinary DP group is [0, 2, 4, 6], while the first expert DP group is [0, 4]. These memberships were verified by executing upstream Megatron's rank-generation functions in isolation.

Matching DP/EDP groups can mask the reduction-group problem: for world_size=32, TP=8, EP=8, ETP=1, PP=2, CP=1, their memberships coincide under that order. This alone does not establish correctness of parameter classification or gradient-norm duplicate filtering. Those effects depend on the installed Megatron/MindSpeed version and need runtime validation.

How to Reproduce / 如何复现
Constructor-level inspection

Run with mcore-bridge and its dependencies installed. Audited source: commit 86a6a5befdb8b2fa995401d6a60cb242f409e57f and the current default branch. This is a source-audit report; the following constructor check has not been executed here, and no NPU training environment/version is being claimed.

from types import SimpleNamespace
import torch
from mcore_bridge.tuners.npu_lora import NpuGroupedLoraLinear

config = SimpleNamespace(
    use_cpu_initialization=True,
    params_dtype=torch.float32,
    tensor_model_parallel_size=2,
    expert_model_parallel_size=2,
    expert_tensor_parallel_size=1,
)
layer = NpuGroupedLoraLinear(
    num_gemms=2, input_size=4, output_size=2,
    config=config, bias=True, is_expert=True,
)
for name, param in layer.named_parameters():
    print(name, hasattr(param, "allreduce"),
          getattr(param, "allreduce", True))
assert all(
    getattr(param, "allreduce", True) is False
    for param in layer.parameters()
)

From the constructor source, the expected inspection result is a missing attribute and a fallback value of True; the assertion is expected to fail.

For an end-to-end check, inspect the actual expert lora_A / lora_B parameters after adapter injection and immediately before DDP/optimizer construction. This also checks whether an external patch compensates for the missing constructor metadata.

Additional Information / 补充信息
Suggested follow-up

Please confirm whether a later MindSpeed/swift step intentionally sets this metadata. Otherwise, initialize the expert adapter parameter attributes consistently with the supported Megatron topology before DDP/optimizer construction, without marking ordinary/shared adapters as expert parameters.

Useful regression coverage would include adapter parameter classification, a configuration with different ordinary DP and expert DP memberships, and inclusion of distinct expert adapter gradients in the global norm.

Related implementation history: #118 introduced this grouped NPU LoRA path; #196 corrected expert checkpoint replica IDs. This report concerns gradient/optimizer metadata rather than checkpoint replica IDs.

Ngôn ngữ chính
Python
Star
105
Fork
54
Merge trung bình
1 ngày 5 giờ
Pull request đã merge (30 ngày)
16

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của modelscope/mcore-bridge

Tất cả issue của modelscope/mcore-bridge

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.