`train_dist.py` error using torch==1.4.0 and torchvision==0.5.0 training on coco dataset TypeError: can't pickle module objects

Đang mở

#356 mở ngày 24/12/2020 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó: 4/5
Thời gian dự kiến: 3-5 ngày
Mức phù hợp với người mới: 30/100

Loại issue: Lỗi
Độ rõ ràng: Cần làm rõ
Mức độ hoạt động: Đình trệ
Công nghệ: python, pytorch
Lĩnh vực: machine-learning

Hướng nghiên cứu

Tái hiện lỗi từ train_dist.py bằng torch==1.4.0, torchvision==0.5.0 và pycocotools==2.0.0 với lệnh COCO đã được báo cáo. Theo dõi quá trình khởi động huấn luyện cho đến khi xác định được module-pickling TypeError; hoàn thành có nghĩa là lệnh bắt đầu huấn luyện COCO mà không gặp lỗi đó.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

with specification below

torch==1.4.0
torchvision==0.5.0
pycocotools==2.0.0

I run

python setup.py install in the working directory

and run

python train_dist.py --dataset coco --model deeplab --backbone resnest50 --aux

having logs like below

3
Namespace(aux=True, aux_weight=0.2, backbone='resnest50', base_size=520, batch_size=2, checkname='default', crop_size=480, dataset='coco', dist_backend='nccl', dist_url='tcp://localhost:12321', epochs=30, eval=False, export=None, ft=False, lr=0.0005, lr_scheduler='poly', model='deeplab', model_zoo=None, momentum=0.9, rank=0, rectify=False, rectify_avg=False, resume=None, se_loss=False, se_weight=0.2, seed=1, start_epoch=0, test_batch_size=16, test_folder=None, test_val=False, train_split='train', weight_decay=0.0001, workers=8, world_size=1)
3
rank: 0 / 3
BaseDataset: base_size 520, crop_size 480
train set
loading annotations into memory...
Done (t=27.11s)
creating index...
index created!
val set
loading annotations into memory...
Done (t=2.57s)
creating index...
index created!
DeepLabV3(
  (pretrained): ResNet(
    (conv1): Sequential(
      (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): DistSyncBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU(inplace=True)
      (3): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (4): DistSyncBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (5): ReLU(inplace=True)
      (6): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    )
    (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
    (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
    (layer1): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): AvgPool2d(kernel_size=1, stride=1, padding=0)
          (1): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (2): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (layer2): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (avd_layer): AvgPool2d(kernel_size=3, stride=2, padding=1)
        (conv2): SplAtConv2d(
          (conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): AvgPool2d(kernel_size=2, stride=2, padding=0)
          (1): Conv2d(256, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (2): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (3): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (layer3): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (avd_layer): AvgPool2d(kernel_size=3, stride=1, padding=1)
        (conv2): SplAtConv2d(
          (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): AvgPool2d(kernel_size=1, stride=1, padding=0)
          (1): Conv2d(512, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (2): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (3): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (4): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (5): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (layer4): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (avd_layer): AvgPool2d(kernel_size=3, stride=1, padding=1)
        (conv2): SplAtConv2d(
          (conv): Conv2d(512, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): AvgPool2d(kernel_size=1, stride=1, padding=0)
          (1): Conv2d(1024, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (2): DistSyncBatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(512, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): DistSyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): SplAtConv2d(
          (conv): Conv2d(512, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), groups=2, bias=False)
          (bn0): DistSyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (fc1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1))
          (bn1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (fc2): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1))
          (rsoftmax): rSoftMax()
        )
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): DistSyncBatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (avgpool): GlobalAvgPool2d()
    (fc): None
  )
  (head): DeepLabV3Head(
    (aspp): ASPP_Module(
      (b0): Sequential(
        (0): Conv2d(2048, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
      )
      (b1): Sequential(
        (0): Conv2d(2048, 256, kernel_size=(3, 3), stride=(1, 1), padding=(12, 12), dilation=(12, 12), bias=False)
        (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
      )
      (b2): Sequential(
        (0): Conv2d(2048, 256, kernel_size=(3, 3), stride=(1, 1), padding=(24, 24), dilation=(24, 24), bias=False)
        (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
      )
      (b3): Sequential(
        (0): Conv2d(2048, 256, kernel_size=(3, 3), stride=(1, 1), padding=(36, 36), dilation=(36, 36), bias=False)
        (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
      )
      (b4): AsppPooling(
        (gap): Sequential(
          (0): AdaptiveAvgPool2d(output_size=1)
          (1): Conv2d(2048, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (2): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (3): ReLU(inplace=True)
        )
      )
      (project): Sequential(
        (0): Conv2d(1280, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU(inplace=True)
        (3): Dropout2d(p=0.5, inplace=False)
      )
    )
    (block): Sequential(
      (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU(inplace=True)
      (3): Dropout(p=0.1, inplace=False)
      (4): Conv2d(256, 21, kernel_size=(1, 1), stride=(1, 1))
    )
  )
  (auxlayer): FCNHead(
    (conv5): Sequential(
      (0): Conv2d(1024, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): DistSyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
      (3): Dropout(p=0.1, inplace=False)
      (4): Conv2d(256, 21, kernel_size=(1, 1), stride=(1, 1))
    )
  )
)
Using poly LR scheduler with warm-up epochs of 0!
Starting Epoch: 0
Total Epoches: 30
3
rank: 2 / 3
BaseDataset: base_size 520, crop_size 480
train set
loading annotations into memory...
Done (t=26.99s)
creating index...
index created!
val set
loading annotations into memory...
Done (t=2.61s)
creating index...
index created!
Using poly LR scheduler with warm-up epochs of 0!
3
rank: 1 / 3
BaseDataset: base_size 520, crop_size 480
train set
loading annotations into memory...
Done (t=27.07s)
creating index...
index created!
val set
loading annotations into memory...
Done (t=2.73s)
creating index...
index created!
Using poly LR scheduler with warm-up epochs of 0!
Traceback (most recent call last):
  File "train_dist.py", line 320, in <module>
    main()
  File "train_dist.py", line 148, in main
    mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception: 

-- Process 2 terminated with the following error:
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/data2/shared/research/img_segmentation/fresh_pytorch_encoding/experiments/segmentation/train_dist.py", line 309, in main_worker
    training(epoch)
  File "/data2/shared/research/img_segmentation/fresh_pytorch_encoding/experiments/segmentation/train_dist.py", line 242, in training
    for i, (image, target) in enumerate(trainloader):
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 719, in __init__
    w.start()
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/user/miniconda3/envs/pytorch_encoding/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle module objects

Ngôn ngữ chính: Python
Star: 2k
Fork: 448
Chỉ số merge pull request: Không có pull request nào được merge trong 30 ngày

Chuẩn bị môi trường

Có Dockerfile hoặc tệp Docker Compose
Không có mẫu pull request
Không có hướng dẫn đóng góp

Bắt đầu từ đâu

Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
Fork repository và làm thay đổi trên một nhánh.
Mở pull request có tham chiếu số hiệu của issue.

Issue khác của zhanghang1989/PyTorch-Encoding

The solution of problem about: no module named 'enclib_cpu' or no module named 'enclib_gpu'
Đang mở
- Có gì không ổn?
Độ khó 3/5 1-2 ngày Mức phù hợp với người mới 30/100

zhanghang1989/PyTorch-Encoding#432 · 2 bình luận · Mở ngày 28/3/2025
Failed to build installable wheels for some pyproject.toml based projects
Đang mở
- Có gì không ổn?
Độ khó 3/5 1-2 ngày Mức phù hợp với người mới 25/100

zhanghang1989/PyTorch-Encoding#430 · 3 bình luận · 2 reaction · Mở ngày 25/9/2024
,
Đang mở
- Có gì không ổn?
Độ khó 5/5 Hơn một tuần Mức phù hợp với người mới 1/100

zhanghang1989/PyTorch-Encoding#429 · Mở ngày 8/9/2024
问题一：print(model）与forward中定义的model不一致，与文章中的结构不一致。问题二：且训练时，无法更新fc层梯度
Đang mở
- Có gì không ổn?
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 25/100

zhanghang1989/PyTorch-Encoding#427 · Mở ngày 9/5/2024
RuntimeError: cudaGetLastError() == cudaSuccess INTERNAL ASSERT FAILED at "/home/msadi002/2023_challenge-main/baseline/utils/lib/gpu/encoding_kernel.cu":233
Đang mở
- Có gì không ổn?
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 25/100

zhanghang1989/PyTorch-Encoding#426 · 1 bình luận · Mở ngày 4/8/2023

Tất cả issue của zhanghang1989/PyTorch-Encoding

Issue tương tự

SlurmSettings.from_mapping truncates numbers, reads "false" as true and accepts a negative signal grace
Đang mở
- Có gì không ổn?
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 76/100

autowarefoundation/autoware_carla_scenario#1485 · Mở ngày 5/11/2026

Maintainer thường phản hồi trong vòng 1 ngày
[automated] Main CI failure issues discard the collected diagnosis
Đang mở
- Có gì không ổn?
needs-area-label

Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 66/100

microsoft/aspire#21778 · Mở ngày 5/11/2026

Maintainer thường phản hồi trong vòng 2 ngày
ci: check that python -m openphysgraspgen runs from the installed wheel
Đang mở
- Có gì không ổn?
enhancement good first issue

Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 84/100

Official-Space-AI/OpenPhysGraspGen#114 · Mở ngày 5/11/2026

Maintainer thường phản hồi trong vòng 2 ngày
New Notion API Changelog Entry: November 4, 2026
Đang mở
- Có gì không ổn?
changelog investigate

Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 62/100

ramnes/notion-sdk-py#441 · Mở ngày 5/11/2026
copy_tree splits same-file errors into characters
Có thể đã có người làm @Gambit-Checkmate đã nhận hôm nay. Đang mở
- Có gì không ổn?
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 68/100

mahmoud/boltons#733 · Mở ngày 5/11/2026

Maintainer thường phản hồi trong vòng 14 ngày

Thêm issue về Python

`train_dist.py` error using torch==1.4.0 and torchvision==0.5.0 training on coco dataset TypeError: can't pickle module objects

Có gì không ổn?

Đánh giá

Hướng nghiên cứu

Mô tả

Chuẩn bị môi trường

Bắt đầu từ đâu

Issue khác của zhanghang1989/PyTorch-Encoding

Có gì không ổn?

Có gì không ổn?

Có gì không ổn?

Có gì không ổn?

Có gì không ổn?

Issue tương tự

Có gì không ổn?

Có gì không ổn?

Có gì không ổn?

Có gì không ổn?

Có gì không ổn?

Nhận issue mới trong hộp thư của bạn