Lightning-AI/pytorch-lightning

Make lazy initialization in plugins more robust

オープン

#7,650 opened on 2021/05/21

 (4 件のコメント) (0 件のリアクション) (1 人の担当者)Python (3,233 件のフォーク)batch import
distributedfeaturegood first issuehelp wantedlet's do it!

Repository metrics

Stars
 (26,687 個のスター)
PR merge metrics
 (平均マージ 9d 15h) (30d で 3 merged PRs)

説明

🚀 Feature

We currently set some attributes like sync_batch_norm, num_nodes, devices etc. lazily in the accelerator connector, so the user does not have to provide them at instantiation.

Example:

Trainer(gpus=4, plugins=DDPPlugin(find_unused_parameters=True))  # plugin may require gpus, num_nodes etc.

# AcceleratorConnector does this:
training_type_plugin.num_nodes = ...

This is fragile. Some attributes may have to be recomputed based on the order in which others are set.

Pitch

Provide one single lazy init method that takes all arguments required. The plugin is responsible for making sure dependencies are resolved in one place:

class DDPPlugin(...):

    def lazy_init(self, **kwargs):
         self.num_nodes = kwargs.get("num_nodes")
         self.num_processes = ...

With lazy_init, the _configure_launcher method (#11643) would become obsolete. It can be merged together into lazy_init.

コントリビューターガイド