`train_dist.py` error using torch==1.4.0 and torchvision==0.5.0 training on coco dataset TypeError: can't pickle module objects

オープン

#356 2020年12月24日に作成コメント 0 件リアクション 0 件担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度: 4/5
見積もり時間: 3〜5日
初心者へのやさしさ: 30/100

issue の種類: バグ
明瞭さ: 説明が足りない
活発さ: 停滞
技術スタック: python, pytorch
領域: machine-learning

調査の方向性

報告された COCO コマンドを使い、torch==1.4.0、torchvision==0.5.0、pycocotools==2.0.0 で train_dist.py の失敗を再現します。module-pickling TypeError を特定するまでトレーニングの起動を追跡します。完了とは、コマンドがそのエラーなしで COCO トレーニングを開始することです。

索引モデルが issue の本文から書いたものです。

説明

with specification below

torch==1.4.0
torchvision==0.5.0
pycocotools==2.0.0

I run

python setup.py install in the working directory

and run

python train_dist.py --dataset coco --model deeplab --backbone resnest50 --aux

having logs like below

3
Namespace(aux=True, aux_weight=0.2, backbone='resnest50', base_size=520, batch_size=2, checkname='default', crop_size=480, dataset='coco', dist_backend='nccl', dist_url='tcp://localhost:12321', epochs=30, eval=False, export=None, ft=False, lr=0.0005, lr_scheduler='poly', model='deeplab', model_zoo=None, momentum=0.9, rank=0, rectify=False, rectify_avg=False, resume=None, se_loss=False, se_weight=0.2, seed=1, start_epoch=0, test_batch_size=16, test_folder=None, test_val=False, train_split='train', weight_decay=0.0001, workers=8, world_size=1)
3
rank: 0 / 3
BaseDataset: base_size 520, crop_size 480
train set
loading annotations into memory...
Done (t=27.11s)
creating index...
index created!
val set
loading annotations into memory...
Done (t=2.57s)
creating index...
index created!
DeepLabV3(
  (pretrained): ResNet(
    (conv1): Sequential(
      (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): DistSyncBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU(inplace=True)
      (3): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (4): DistSyncBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (5): ReLU(inplace=True)
      (6): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    )
    (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
    (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
    (layer1): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): AvgPool2d(kernel_size=1, stride=1, padding=0)
          (1): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (2): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (layer2): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (avd_layer): AvgPool2d(kernel_size=3, stride=2, padding=1)
        (conv2): SplAtConv2d(
          (conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): AvgPool2d(kernel_size=2, stride=2, padding=0)
          (1): Conv2d(256, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (2): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (3): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (layer3): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (avd_layer): AvgPool2d(kernel_size=3, stride=1, padding=1)
        (conv2): SplAtConv2d(
          (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): AvgPool2d(kernel_size=1, stride=1, padding=0)
          (1): Conv2d(512, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (2): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (3): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (4): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (5): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (layer4): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (avd_layer): AvgPool2d(kernel_size=3, stride=1, padding=1)
        (conv2): SplAtConv2d(
          (conv): Conv2d(512, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): AvgPool2d(kernel_size=1, stride=1, padding=0)
          (1): Conv2d(1024, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (2): DistSyncBatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(512, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(512, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (avgpool): GlobalAvgPool2d()
    (fc): None
  )
  (head): DeepLabV3Head(
    (aspp): ASPP_Module(
      (b0): Sequential(
        (0): Conv2d(2048, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
      )
      (b1): Sequential(
        (0): Conv2d(2048, 256, kernel_size=(3, 3), stride=(1, 1), padding=(12, 12), dilation=(12, 12), bias=False)
        (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
      )
      (b2): Sequential(
        (0): Conv2d(2048, 256, kernel_size=(3, 3), stride=(1, 1), padding=(24, 24), dilation=(24, 24), bias=False)
        (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
      )
      (b3): Sequential(
        (0): Conv2d(2048, 256, kernel_size=(3, 3), stride=(1, 1), padding=(36, 36), dilation=(36, 36), bias=False)
        (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
      )
      (b4): AsppPooling(
        (gap): Sequential(
          (0): AdaptiveAvgPool2d(output_size=1)
          (1): Conv2d(2048, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (2): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (3): ReLU(inplace=True)
        )
      )
      (project): Sequential(
        (0): Conv2d(1280, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
        (3): Dropout2d(p=0.5, inplace=False)
      )
    )
    (block): Sequential(
      (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU(inplace=True)
      (3): Dropout(p=0.1, inplace=False)
      (4): Conv2d(256, 21, kernel_size=(1, 1), stride=(1, 1))
    )
  )
  (auxlayer): FCNHead(
    (conv5): Sequential(
      (0): Conv2d(1024, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
      (3): Dropout(p=0.1, inplace=False)
      (4): Conv2d(256, 21, kernel_size=(1, 1), stride=(1, 1))
    )
  )
)
Using poly LR scheduler with warm-up epochs of 0!
Starting Epoch: 0
Total Epoches: 30
3
rank: 2 / 3
BaseDataset: base_size 520, crop_size 480
train set
loading annotations into memory...
Done (t=26.99s)
creating index...
index created!
val set
loading annotations into memory...
Done (t=2.61s)
creating index...
index created!
Using poly LR scheduler with warm-up epochs of 0!
3
rank: 1 / 3
BaseDataset: base_size 520, crop_size 480
train set
loading annotations into memory...
Done (t=27.07s)
creating index...
index created!
val set
loading annotations into memory...
Done (t=2.73s)
creating index...
index created!
Using poly LR scheduler with warm-up epochs of 0!
Traceback (most recent call last):
  File "train_dist.py", line 320, in <module>
    main()
  File "train_dist.py", line 148, in main
    mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception: 

-- Process 2 terminated with the following error:
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/data2/shared/research/img_segmentation/fresh_pytorch_encoding/experiments/segmentation/train_dist.py", line 309, in main_worker
    training(epoch)
  File "/data2/shared/research/img_segmentation/fresh_pytorch_encoding/experiments/segmentation/train_dist.py", line 242, in training
    for i, (image, target) in enumerate(trainloader):
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 719, in __init__
    w.start()
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle module objects

主要言語: Python
スター: 2k
フォーク: 448
PR マージ指標: 30日以内にマージされた PR はありません

環境構築

Dockerfile または Docker Compose ファイルあり
プルリクエストのテンプレートなし
コントリビューションガイドなし

はじめの一歩

issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
リポジトリをフォークし、ブランチを切って変更します。
issue 番号を参照したプルリクエストを送ります。

zhanghang1989/PyTorch-Encoding のほかの issue

The solution of problem about: no module named 'enclib_cpu' or no module named 'enclib_gpu'
オープン
- 何か問題がありますか？
難易度 3/5 1〜2日初心者へのやさしさ 30/100

zhanghang1989/PyTorch-Encoding#432 · コメント 2 件 · 2025年3月28日に作成
Failed to build installable wheels for some pyproject.toml based projects
オープン
- 何か問題がありますか？
難易度 3/5 1〜2日初心者へのやさしさ 25/100

zhanghang1989/PyTorch-Encoding#430 · コメント 3 件 · リアクション 2 件 · 2024年9月25日に作成
,
オープン
- 何か問題がありますか？
難易度 5/5 1週間以上初心者へのやさしさ 1/100

zhanghang1989/PyTorch-Encoding#429 · 2024年9月8日に作成
问题一：print(model）与forward中定义的model不一致，与文章中的结构不一致。问题二：且训练时，无法更新fc层梯度
オープン
- 何か問題がありますか？
難易度 4/5 3〜5日初心者へのやさしさ 25/100

zhanghang1989/PyTorch-Encoding#427 · 2024年5月9日に作成
RuntimeError: cudaGetLastError() == cudaSuccess INTERNAL ASSERT FAILED at "/home/msadi002/2023_challenge-main/baseline/utils/lib/gpu/encoding_kernel.cu":233
オープン
- 何か問題がありますか？
難易度 4/5 3〜5日初心者へのやさしさ 25/100

zhanghang1989/PyTorch-Encoding#426 · コメント 1 件 · 2023年8月4日に作成

zhanghang1989/PyTorch-Encoding の issue をすべて見る

似ている issue

Windows: tool calls after the first in a turn fail with "Effect claim could not be persisted durably" (effect journal opened in text mode)
オープン
- 何か問題がありますか？
bug ready for review

難易度 2/5 1〜3時間初心者へのやさしさ 86/100

odysseus-dev/odysseus#6641 · 2026年10月9日に作成

メンテナーはふだん 1 日以内に返信
[bug] Color primario se cambio de naranja a zul
オープン
- 何か問題がありますか？
bug

難易度 2/5 1〜3時間初心者へのやさしさ 76/100

happypawspillaro/happypaws#78 · 2026年10月9日に作成

メンテナーはふだん 4 日以内に返信
xAI: replayed web search tool calls lose their function name (`open_page`, `browse_page`, … → `web_search`)
オープン
- 何か問題がありますか？
pydanty:is-working

難易度 2/5 1〜3時間初心者へのやさしさ 82/100

pydantic/pydantic-ai#10020 · 2026年10月9日に作成

メンテナーはふだん 1 日以内に返信
[Bug][zos_copy] Silent false success when copying PDS/GDG to USS directory where dataset-name path exists as a file
オープン
- 何か問題がありますか？
Bug

難易度 2/5 1〜3時間初心者へのやさしさ 78/100

ansible-collections/ibm_zos_core#2650 · 2026年10月8日に作成
[vLLM] Watermarking unit tests out of sync with vLLM APIs at the current pin
オープン
- 何か問題がありますか？
hw: pvc tests: vllm vllm

難易度 2/5 1〜3時間初心者へのやさしさ 68/100

intel/intel-xpu-backend-for-triton#8362 · 2026年10月8日に作成

メンテナーはふだん 1 日以内に返信

Python の issue をもっと見る

`train_dist.py` error using torch==1.4.0 and torchvision==0.5.0 training on coco dataset TypeError: can't pickle module objects

何か問題がありますか？

評価

調査の方向性

説明

環境構築

はじめの一歩

zhanghang1989/PyTorch-Encoding のほかの issue

何か問題がありますか？

何か問題がありますか？

何か問題がありますか？

何か問題がありますか？

何か問題がありますか？

似ている issue

何か問題がありますか？

何か問題がありますか？

何か問題がありますか？

何か問題がありますか？

何か問題がありますか？

新しい issue をメールで受け取る