Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Add features for Qwen2_moe support

Abierto
  • ¿Algo no está bien?
#164 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
22/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Estancado
Stack tecnológico
python
Área
machine-learning

Línea de trabajo

The issue names no files or tests, so start by locating where the Qwen2 MoE model config and decoder layers are defined in the repository. Add the parameters decoder_sparse_step, moe_intermediate_size, shared_expert_intermediate_size, norm_topk_prob and mlp_only_layers, then use the layer choice to switch between dense MLP and sparse MoE blocks. Done means these options are honored, covered by tests, and documented.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement

🎯 Goal (What & Why)

The following additional parameters need to be added and their functionality implemented to support Qwen2_moe models:

  • decoder_sparse_step (int, optional, defaults to 1):
    The frequency of the MoE layer.
  • moe_intermediate_size (int, optional, defaults to 1408):
    Intermediate size of the routed expert.
  • shared_expert_intermediate_size (int, optional, defaults to 5632):
    Intermediate size of the shared expert.
  • norm_topk_prob (bool, optional, defaults to False):
    Whether to normalize the topk probabilities.
  • mlp_only_layers (List[int], optional, defaults to []):
    Indicate which layers use dense MLP rather than Sparse Moe blocks
    The list contains layer index, from 0 to num_layers-1 if we have num_layers layers
    If mlp_only_layers is empty, decoder_sparse_step is used to determine the sparsity.

Additionally add_linear_bias=only_attn_qkv needs to be supported then using MLP MoE blocks.

This is needed for #135

🚀 Execution Plan

**Step 1: Changes in MLP Moe
  • Implement custom intermediate sizes for shared and non shared Experts
  • add support for norm_topk_prob
**Step 2: Changes in layer architecture
  • Implement decoder_sparse_step and mlp_only_layers

📌 Acceptance Criteria (Must-Haves for Completion)

  • The feature must be functional and tested.
  • The implementation must be documented in practical terms.
  • The PR must include a performance/impact summary.
  • No refactors unless directly necessary for feature completion.

🛠️ Project Management

  • Assign the project to the Fast-LLM project.
  • Set the Estimate field (in days) in the GitHub project.
  • Use the Size field to categorize the PR size (Small/Medium/Large).
  • Assign an owner when opening the issue.
Lenguaje dominante
Python
Estrellas
337
Forks
52
Sin PR fusionados en 30 d

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ServiceNow/Fast-LLM

Todos los issues de ServiceNow/Fast-LLM

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.