Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[BUG] NPU grouped expert LoRA parameters lack expert-parallel allreduce metadata

Aperta
  • Qualcosa non va?
#229 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
42/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
distributed-systems, machine-learning

Direzione di ricerca

Start in src/mcore_bridge/tuners/npu_lora.py, in the NpuGroupedLoraLinear constructor where weight{i} and bias{i} are registered without an allreduce attribute, and check how is_expert is passed from src/mcore_bridge/tuners/lora.py. Run the constructor snippet from the issue to confirm the attribute is missing; this needs a Megatron/MindSpeed environment to be meaningful. Done means expert adapter parameters report allreduce=False while ordinary adapters are unchanged, and a test covers that classification.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug
Checklist / 检查清单
  • I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。
Bug Description / Bug 描述
Summary

The NPU grouped LoRA path appears to create per-expert adapter parameters without the parameter-level metadata Megatron uses to select the expert topology.

NpuGroupedLoraLinear stores self.is_expert, but registers weight{i} and optional bias{i} as plain nn.Parameter objects without setting param.allreduce. Megatron consumers fall back to getattr(param, "allreduce", True), so these parameters can be classified as ordinary dense parameters even when the module represents independent local experts.

Validation scope: this report is based on a source audit. I have not reproduced an NPU training failure or a model-quality regression. The constructor inspection below is provided for reproduction in an installed environment; it has not been executed here. An external MindSpeed patch could affect the final runtime behavior.

Source evidence
  • Constructor at commit 86a6a5b: module-level is_expert is retained, but the newly registered parameters receive no allreduce attribute. The current default-branch constructor has the same behavior.
  • LoRA construction path: the NPU grouped branch constructs both LoRA A and B with NpuGroupedLoraLinear(..., is_expert=self.is_expert).
  • Expert checkpoint mapping assigns separate global expert indices, so these adapters are per-expert parameters.
  • I could not find a subsequent adapter-wide setter for this attribute in the audited mcore-bridge path.

Relevant Megatron consumers:

Expected behavior and potential impact

For this is_expert=True, EP=2, ETP=1 case, adapter parameters should receive the expert-topology metadata, including allreduce=False, consistently with Megatron's expert layers. This still synchronizes replicas of the same expert through expert DP.

If nothing sets the metadata later, ordinary DP selection can synchronize parameters belonging to different experts when the two topologies differ. For example, with world_size=8, TP=2, EP=4, ETP=1, PP=1, CP=1 and rank order tp-cp-ep-dp-pp, the first ordinary DP group is [0, 2, 4, 6], while the first expert DP group is [0, 4]. These memberships were verified by executing upstream Megatron's rank-generation functions in isolation.

Matching DP/EDP groups can mask the reduction-group problem: for world_size=32, TP=8, EP=8, ETP=1, PP=2, CP=1, their memberships coincide under that order. This alone does not establish correctness of parameter classification or gradient-norm duplicate filtering. Those effects depend on the installed Megatron/MindSpeed version and need runtime validation.

How to Reproduce / 如何复现
Constructor-level inspection

Run with mcore-bridge and its dependencies installed. Audited source: commit 86a6a5befdb8b2fa995401d6a60cb242f409e57f and the current default branch. This is a source-audit report; the following constructor check has not been executed here, and no NPU training environment/version is being claimed.

from types import SimpleNamespace
import torch
from mcore_bridge.tuners.npu_lora import NpuGroupedLoraLinear

config = SimpleNamespace(
    use_cpu_initialization=True,
    params_dtype=torch.float32,
    tensor_model_parallel_size=2,
    expert_model_parallel_size=2,
    expert_tensor_parallel_size=1,
)
layer = NpuGroupedLoraLinear(
    num_gemms=2, input_size=4, output_size=2,
    config=config, bias=True, is_expert=True,
)
for name, param in layer.named_parameters():
    print(name, hasattr(param, "allreduce"),
          getattr(param, "allreduce", True))
assert all(
    getattr(param, "allreduce", True) is False
    for param in layer.parameters()
)

From the constructor source, the expected inspection result is a missing attribute and a fallback value of True; the assertion is expected to fail.

For an end-to-end check, inspect the actual expert lora_A / lora_B parameters after adapter injection and immediately before DDP/optimizer construction. This also checks whether an external patch compensates for the missing constructor metadata.

Additional Information / 补充信息
Suggested follow-up

Please confirm whether a later MindSpeed/swift step intentionally sets this metadata. Otherwise, initialize the expert adapter parameter attributes consistently with the supported Megatron topology before DDP/optimizer construction, without marking ordinary/shared adapters as expert parameters.

Useful regression coverage would include adapter parameter classification, a configuration with different ordinary DP and expert DP memberships, and inclusion of distinct expert adapter gradients in the global norm.

Related implementation history: #118 introduced this grouped NPU LoRA path; #196 corrected expert checkpoint replica IDs. This report concerns gradient/optimizer metadata rather than checkpoint replica IDs.

Lingua principale
Python
Stelle
104
Fork
49
Merge medio
1g 18h
PR unite (30g)
21

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di modelscope/mcore-bridge

Tutte le issue di modelscope/mcore-bridge

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.