IBM/ModuleFormer
View on GitHubModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters.
- Stars
- 225
- Forks
- 12
- Open beginner issues
- 0
- Indexed issues
- 3
- Dominant language
- Python
- License
- Apache-2.0
- Last GitHub push
- Jan 2, 2024
- Latest indexed
- Sep 17, 2026
- Contributing guide
- No contributing guide
- Code of conduct
- No code of conduct
- Beginner labels
- No beginner labels indexed
- PR merge metrics
- No merged PRs in 30d
Loading issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
IBM/ModuleFormer#13 ·
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
IBM/ModuleFormer#11 ·
-
Length exploration Open
Difficulty 5/5 Over a week Newbie friendliness 20/100
IBM/ModuleFormer#9 ·