help wantedinactive
Description
Platform: RunPod GPU: 1x A100
Steps to Reproduce
from unsloth import FastLanguageModel, GRPOTrainer, QLoRAConfig
from datasets import load_dataset
import torch
Observed Error
Flash Attention 2 issues detected, tried to fall back to Xformers. Then encountered:
RuntimeError: Unsloth: Please file a bug report! Error patching SFTTrainer
Full Error Traceback
---------------------------------------------------------------------------
SyntaxError Traceback (most recent call last)
File ~/miniconda3/envs/py3.11/lib/python3.11/site-packages/unsloth/tokenizer_utils.py:1037
1036 try:
-> 1037 exec(trainer_text, globals())
1038 except:
SyntaxError: invalid syntax (<string>, line 4)
During handling of the above exception, another exception occurred:
RuntimeError Traceback (most recent call last)
Cell In[1], line 6
3 get_ipython().system('apt-get update && apt-get install -y iverilog')
5 # Import libraries
----> 6 from unsloth import FastLanguageModel, GRPOTrainer, QLoRAConfig
7 from datasets import load_dataset
8 import torch
File ~/miniconda3/envs/py3.11/lib/python3.11/site-packages/unsloth/__init__.py:174
171 raise ImportError("Unsloth: Please install unsloth_zoo via `pip install unsloth-zoo`")
172 pass
--> 174 from .models import *
175 from .save import *
176 from .chat_templates import *
File ~/miniconda3/envs/py3.11/lib/python3.11/site-packages/unsloth/models/__init__.py:16
1 # Copyright 2023-present Daniel Han-Chen & the Unsloth team. All rights reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...) 12 # See the License for the specific language governing permissions and
13 # limitations under the License.
---> 16 from .granite import FastGraniteModel
17 from .loader import FastLanguageModel, FastVisionModel
18 from .llama import FastLlamaModel
File ~/miniconda3/envs/py3.11/lib/python3.11/site-packages/unsloth/models/granite.py:15
1 # Copyright 2023-present Daniel Han-Chen & the Unsloth team. All rights reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...) 12 # See the License for the specific language governing permissions and
13 # limitations under the License.
---> 15 from .llama import *
16 import os
17 from ._utils import __version__
File ~/miniconda3/envs/py3.11/lib/python3.11/site-packages/unsloth/models/llama.py:32
28 from transformers.modeling_attn_mask_utils import (
29 _prepare_4d_causal_attention_mask_for_sdpa,
30 )
31 from ..kernels import *
---> 32 from ..tokenizer_utils import *
33 if HAS_FLASH_ATTENTION:
34 from flash_attn import flash_attn_func
File ~/miniconda3/envs/py3.11/lib/python3.11/site-packages/unsloth/tokenizer_utils.py:1039
1037 exec(trainer_text, globals())
1038 except:
-> 1039 raise RuntimeError(f"Unsloth: Please file a bug report! Error patching {trainer_name}")
1040 exec(f"trl.trainer.{trainer_name} = Unsloth{trainer_name}", globals())
1041 pass
RuntimeError: Unsloth: Please file a bug report! Error patching SFTTrainer
Unsloth Output
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
Unsloth: Your Flash Attention 2 installation seems to be broken?
A possible explanation is you have a new CUDA version which isn't
yet compatible with FA2? Please file a ticket to Unsloth or FA2.
We shall now use Xformers instead, which does not have any performance hits!
We found this negligible impact by benchmarking on 1x A100.
🦥 Unsloth Zoo will now patch everything to make training faster!
Additional Information
- Using GRPOTrainer module for training
- The error occurs at the import stage