unslothai/unsloth

[Bug] Llama-3.1-8B Not Supported in Unsloth 2024.09.post2

Open

#2,503 opened on May 9, 2025

View on GitHub
 (7 comments) (1 reaction) (0 assignees)Python (64,271 stars) (5,658 forks)batch import
help wanted

Description

When attempting to load meta-llama/Llama-3.1-8B with Unsloth version 2024.09.post2, I receive a NotImplementedError stating that the model is not supported and suggesting an upgrade to the latest Unsloth version. However, upgrading Unsloth is not feasible in my environment due to strict transformers version constraints..

Is there a known workaround for this issue, or an official compatibility matrix for Unsloth and transformers versions? Since my training runs inside a managed SageMaker Docker container, I am unable to patch the package source code directly (as suggested in #1726).

Any advice for resolving this without upgrading Unsloth or transformers would be greatly appreciated. Thank you!

Contributor guide

Tech stack
pythonpytorch
Domain
machine learning
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Pythontransformers libraryUnsloth basicsSageMaker
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50
Research direction
Investigate the Unsloth source code, particularly the model loading and registration logic in unsloth/models/ to understand why Llama 3.1-8B is not supported in the older version. Review linked issue #1726 for workarounds involving manual patching of package files. Since environment constraints prevent upgrading, explore the possibility of pinning a compatible transformers version or creating a custom Docker image with the required dependencies.