Prodigy Optimizer.

Hello!

First of all, I would like to thank the devs for their effort in maintaining this repository.

I would like to suggest an update.

Update the optimizer to Prodigy.

Prodigy is a new optimizer that converges much faster and easier than Adam. It is the direct evolution of Dadptation.

Prodigy has been widely used in Stable Diffusion now for delivering faster results and allowing the choice of LR to be automatic.

I implemented PRODIGY in my local environment and it worked very well.

It's not very difficult to implement and modify.

In this repository, you can find the paper and the requirements to use Prodigy.

From optimize args I used the standards we are currently using in SD. "decouple=True" "weight_decay=0.01" "d_coef=2" "use_bias_correction=True" "safeguard_warmup=True"

I really think this can speed up model training and deliver better results. It would also be recommended to trigger an early stop system to lock the training at the moment it starts to want to overfit.

I also implemented in my local env a wandb integration, but for that i removed the tensorboard. So i dont know if it valid for here, but i like more Wandb since we can acess it from anywhere.

https://github.com/konstmish/prodigy

So that's it.

Any doubts you can contact me!

Have a good day!