enhancementhelp wantedneeds triage
仓库指标
- Star
- (29,107 star)
- PR 合并指标
- (30 天内没有已合并 PR)
描述
🚀 Feature Request
Support BF16 with FSDP!
Motivation
Training with BF16 is usually more stable than fp16, which is very important when we want to train large models with FSDP. Additionally, many models (e.g. T5) are trained with BF16 and if we want to continue training them via Fairseq with mixed-precisio, using fp16 will result in NaNs.
Additional context
Currently FullyShardedDataParallel is not compatible with bf16 (as can be seen here). It will be really helpful to support it.