Allow multiple parallel wandb runs by using/initializing wandb not globally but locally

🚀 Feature Request

Instead of using wandb in a global manner (with e.g. wandb.init or wandb.config.update) use it locally. This means, store the initialized run as an attribute of the WandBProgressBarWrapper via (line 403 in WandBProgressBarWrapper):

self.run = wandb.init(project=wandb_project, reinit=False, name=run_name)

This way, other code outside of the library, in my case my own code, is able to still use wandb independent from fairseq, i.e. initialize other runs and log values to these runs.

Motivation

When using the fairseq library/code in my own experiments, where I train multiple models within one run, it is not possible for me to log data to different wandb runs. Wandb throws exceptions, that the global settings like its config attribute can not be updated:

wandb.sdk.lib.config_util.ConfigError: Attempted to change value of key "task" from domain to [...]. If you really want to do this, pass allow_val_change=True to config.update()

This happens, because fairseq uses wandb in a global manner and along with this sets flags like reinit in wandb.init()or allow_val_change in wandb.config.update() to False.

Pitch

Be able to instantiate multiple wandb runs and log to them independently alongside with using the fairseq library in my code. The following, for example, should be possible even when passing a value for the parameter --wandb-project to fairseq:

run1 = wandb.init(...)
run2 = wandb.init(...)
run3 = wandb.init(...)

run1.log({'loss': 0.132})
run2.log({'fooBar': 238})

Alternatives

Additional context