Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[BUG] NPU grouped expert LoRA parameters lack expert-parallel allreduce metadata

Abierto
  • ¿Algo no está bien?
#229 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
42/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python, pytorch
Área
distributed-systems, machine-learning

Línea de trabajo

Start in src/mcore_bridge/tuners/npu_lora.py, in the NpuGroupedLoraLinear constructor where weight{i} and bias{i} are registered without an allreduce attribute, and check how is_expert is passed from src/mcore_bridge/tuners/lora.py. Run the constructor snippet from the issue to confirm the attribute is missing; this needs a Megatron/MindSpeed environment to be meaningful. Done means expert adapter parameters report allreduce=False while ordinary adapters are unchanged, and a test covers that classification.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug
Checklist / 检查清单
  • I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。
Bug Description / Bug 描述
Summary

The NPU grouped LoRA path appears to create per-expert adapter parameters without the parameter-level metadata Megatron uses to select the expert topology.

NpuGroupedLoraLinear stores self.is_expert, but registers weight{i} and optional bias{i} as plain nn.Parameter objects without setting param.allreduce. Megatron consumers fall back to getattr(param, "allreduce", True), so these parameters can be classified as ordinary dense parameters even when the module represents independent local experts.

Validation scope: this report is based on a source audit. I have not reproduced an NPU training failure or a model-quality regression. The constructor inspection below is provided for reproduction in an installed environment; it has not been executed here. An external MindSpeed patch could affect the final runtime behavior.

Source evidence
  • Constructor at commit 86a6a5b: module-level is_expert is retained, but the newly registered parameters receive no allreduce attribute. The current default-branch constructor has the same behavior.
  • LoRA construction path: the NPU grouped branch constructs both LoRA A and B with NpuGroupedLoraLinear(..., is_expert=self.is_expert).
  • Expert checkpoint mapping assigns separate global expert indices, so these adapters are per-expert parameters.
  • I could not find a subsequent adapter-wide setter for this attribute in the audited mcore-bridge path.

Relevant Megatron consumers:

Expected behavior and potential impact

For this is_expert=True, EP=2, ETP=1 case, adapter parameters should receive the expert-topology metadata, including allreduce=False, consistently with Megatron's expert layers. This still synchronizes replicas of the same expert through expert DP.

If nothing sets the metadata later, ordinary DP selection can synchronize parameters belonging to different experts when the two topologies differ. For example, with world_size=8, TP=2, EP=4, ETP=1, PP=1, CP=1 and rank order tp-cp-ep-dp-pp, the first ordinary DP group is [0, 2, 4, 6], while the first expert DP group is [0, 4]. These memberships were verified by executing upstream Megatron's rank-generation functions in isolation.

Matching DP/EDP groups can mask the reduction-group problem: for world_size=32, TP=8, EP=8, ETP=1, PP=2, CP=1, their memberships coincide under that order. This alone does not establish correctness of parameter classification or gradient-norm duplicate filtering. Those effects depend on the installed Megatron/MindSpeed version and need runtime validation.

How to Reproduce / 如何复现
Constructor-level inspection

Run with mcore-bridge and its dependencies installed. Audited source: commit 86a6a5befdb8b2fa995401d6a60cb242f409e57f and the current default branch. This is a source-audit report; the following constructor check has not been executed here, and no NPU training environment/version is being claimed.

from types import SimpleNamespace
import torch
from mcore_bridge.tuners.npu_lora import NpuGroupedLoraLinear

config = SimpleNamespace(
    use_cpu_initialization=True,
    params_dtype=torch.float32,
    tensor_model_parallel_size=2,
    expert_model_parallel_size=2,
    expert_tensor_parallel_size=1,
)
layer = NpuGroupedLoraLinear(
    num_gemms=2, input_size=4, output_size=2,
    config=config, bias=True, is_expert=True,
)
for name, param in layer.named_parameters():
    print(name, hasattr(param, "allreduce"),
          getattr(param, "allreduce", True))
assert all(
    getattr(param, "allreduce", True) is False
    for param in layer.parameters()
)

From the constructor source, the expected inspection result is a missing attribute and a fallback value of True; the assertion is expected to fail.

For an end-to-end check, inspect the actual expert lora_A / lora_B parameters after adapter injection and immediately before DDP/optimizer construction. This also checks whether an external patch compensates for the missing constructor metadata.

Additional Information / 补充信息
Suggested follow-up

Please confirm whether a later MindSpeed/swift step intentionally sets this metadata. Otherwise, initialize the expert adapter parameter attributes consistently with the supported Megatron topology before DDP/optimizer construction, without marking ordinary/shared adapters as expert parameters.

Useful regression coverage would include adapter parameter classification, a configuration with different ordinary DP and expert DP memberships, and inclusion of distinct expert adapter gradients in the global norm.

Related implementation history: #118 introduced this grouped NPU LoRA path; #196 corrected expert checkpoint replica IDs. This report concerns gradient/optimizer metadata rather than checkpoint replica IDs.

Lenguaje dominante
Python
Estrellas
104
Forks
49
Merge medio
1 d 18 h
PR fusionados (30 d)
21

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de modelscope/mcore-bridge

Todos los issues de modelscope/mcore-bridge

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.