Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Per-example & Phatgoose routing

Abierto
  • ¿Algo no está bien?
#181 0 comentarios 1 reacción 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Estancado
Stack tecnológico
python
Área
machine-learning

Línea de trabajo

Start with the MixtureOfExpertMLP class and its top-k routing code, where the issue asks for a new _per_example_topk_routing function. The Phatgoose part needs a per-expert sigmoid gate and separately learned expert embeddings, which the issue does not spell out, so read the linked papers first. Done means per-example routing and the Phatgoose router both run and have tests, and the behaviour is documented. No tests or entry points are named, so find the existing MoE tests before writing new ones.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement

🎯 Goal (What & Why)

When training experts in the so-called "embarrassingly parallel fashion", I observed that per-example oracle routing performs better (each sample is sent to its dedicated expert that is assumed to be known) than full MoE fine-tuning (like BTX) as well as training only the per-token router.

This motivates the need to experiment with:

  • per-example routing, where router routes based on aggregated token view per sample, each token in a sample receives the same routing weights
  • implementation of Phatgoose type of router (per token, but expert embeddings are learned independently)

🚀 Execution Plan

Step 1: What is the smallest working version?

Implement a new _per_example_topk_routing function in the MixtureOfExpertMLP class for per-example routing.

As for Phatgoose, we also need to implement a sigmoid gate for each expert, the routing embeddings in this case are learned independently for each expert on its own dedicated expert dataset. At inference time, we use the learned expert embeddings with a standard per-token router.

📌 Acceptance Criteria (Must-Haves for Completion)

  • The feature must be functional and tested.
  • The implementation must be documented in practical terms.
  • No refactors unless directly necessary for feature completion.

🛠️ Project Management

  • Assign the project to the Fast-LLM project.
  • Set the Estimate field (in days) in the GitHub project.
  • Use the Size field to categorize the PR size (Small/Medium/Large).
  • Assign an owner when opening the issue.
Lenguaje dominante
Python
Estrellas
337
Forks
52
Sin PR fusionados en 30 d

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ServiceNow/Fast-LLM

Todos los issues de ServiceNow/Fast-LLM

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.