Issues du dépôt

MambaDecoderBlock is a novel decoder architecture that replaces traditional self-attention mechanisms with Mamba state space models, augmented by Mixture of Experts (MoE) layers.