concat 2 diff types dataset train ,val and test issue

Prerequisite

• I have searched the existing and past issues but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version.

🐞 Describe the bug

I got 2 dataset Dataset_A have 15 classes with COCO format Dataset_B have 20 classes with VOC format

I followed the format in https://mmdetection.readthedocs.io/en/latest/tutorials/customize_dataset.html#concatenate-dataset note part to create a concated dataset and the config like this:

img_norm_cfg = dict(
          mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
      train_pipeline = [
          dict(type='LoadImageFromFile'),
          dict(type='LoadAnnotations', with_bbox=True),
          dict(type='Resize', img_scale=(1024, 1024), keep_ratio=True),
          dict(type='RandomFlip', flip_ratio=0.5),
          dict(type='Normalize', **img_norm_cfg),
          dict(type='Pad', size_divisor=32),
          dict(type='DefaultFormatBundle'),
          dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']),
      ]
      test_pipeline = [
          dict(type='LoadImageFromFile'),
          dict(
              type='MultiScaleFlipAug',
              img_scale=(1024, 1024),
              flip=False,
              transforms=[
                  dict(type='Resize', keep_ratio=True),
                  dict(type='RandomFlip'),
                  dict(type='Normalize', **img_norm_cfg),
                  dict(type='Pad', size_divisor=32),
                  dict(type='ImageToTensor', keys=['img']),
                  dict(type='Collect', keys=['img']),
              ])
      ]
      dota_train_dict=dict(
          type='RepeatDataset',
          times=1,
          dataset=dict(
              type='CocoDataset',
              ann_file='ann_path/train_ann.json',
              img_prefix=img_prefix/images/',
              pipeline=train_pipeline
          )
      )
      dota_val_dict=dict(
              type='CocoDataset',
              ann_file='ann_path/val_ann.json',
              img_prefix=img_prefix/images/',
              pipeline=test_pipeline    
      )
      dota_test_dict=dict(
              type='CocoDataset',
              ann_file='ann_path/val_ann.json',
              img_prefix=img_prefix/images/',
              pipeline=test_pipeline
           
      )
      mar20_train_dict=dict(
          type='RepeatDataset',
          times=3,
          dataset=dict(
              type='VOCDataset',
              ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
              img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
              pipeline=train_pipeline
          )
      )
      # mar20_val_dict=dict(
      #     dataset=dict(
      #         type='VOCDataset',
      #        ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
      #        img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
      #         pipeline=test_pipeline
      #     )    
      # )
      # mar20_test_dict=dict(
      #     dataset=dict(
      #         type='VOCDataset',
      #        ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
      #        img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
      #         pipeline=test_pipeline
      #     )    
      # )
      data = dict(
          samples_per_gpu=4,
          workers_per_gpu=2,
          train=[dota_train_dict,mar20_train_dict],
          val=dota_val_dict,
          test=dota_test_dict
          )
      evaluation = dict(interval=1, metric='bbox')

I try to train this concat in a normal faster rcnn r50 fpn network. so i set the num_classes to a+b which is 35 here

then i change both /mmdet/datasets/coco.py and /mmdet/datasets/voc.py file CLASSES part to which also num = 35

CASE1: when I started training, everything was normal ,but i will got issue when this 1st training process and the 1st val process finished.

2022-09-27 15:04:44,246 - mmdet - INFO - Epoch [1][1850/1898]   lr: 1.000e-04, eta: 3:05:32, time: 0.528, data_time: 0.022, memory: 8168, loss_rpn_cls: 0.1822, loss_rpn_bbox: 0.0940, loss_cls: 0.3737, acc: 91.3897, loss_bbox: 0.2509, loss: 0.9008
2022-09-27 15:05:16,011 - mmdet - INFO - Saving checkpoint at 1 epochs
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 10131/10130, 42.3 task/s, elapsed: 240s, ETA:     0s

Traceback (most recent call last):
  File "/home/xxxxx/xxxxx/mmdetection/./tools/train.py", line 185, in <module>
    main()
  File "/home/xxxxx/xxxxx/mmdetection/./tools/train.py", line 174, in main
    train_detector(
  File "/home/xxxxx/xxxxx/mmdetection/mmdet/apis/train.py", line 203, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/home/xxxxx/anaconda3/envs/openmmlab/lib/python3.9/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/xxxxx/anaconda3/envs/openmmlab/lib/python3.9/site-packages/mmcv/runner/epoch_based_runner.py", line 54, in train
    self.call_hook('after_train_epoch')
  File "/home/xxxxx/anaconda3/envs/openmmlab/lib/python3.9/site-packages/mmcv/runner/base_runner.py", line 307, in call_hook
    getattr(hook, fn_name)(self)
  File "/home/xxxxx/anaconda3/envs/openmmlab/lib/python3.9/site-packages/mmcv/runner/hooks/evaluation.py", line 267, in after_train_epoch
    self._do_evaluate(runner)
  File "/home/xxxxx/xxxxx/mmdetection/mmdet/core/evaluation/eval_hooks.py", line 123, in _do_evaluate
    key_score = self.evaluate(runner, results)
  File "/home/xxxxx/anaconda3/envs/openmmlab/lib/python3.9/site-packages/mmcv/runner/hooks/evaluation.py", line 361, in evaluate
    eval_res = self.dataloader.dataset.evaluate(
  File "/home/xxxxx/xxxxx/mmdetection/mmdet/datasets/coco.py", line 445, in evaluate
    result_files, tmp_dir = self.format_results(results, jsonfile_prefix)
  File "/home/xxxxx/xxxxx/mmdetection/mmdet/datasets/coco.py", line 390, in format_results
    result_files = self.results2json(results, jsonfile_prefix)
  File "/home/xxxxx/xxxxx/mmdetection/mmdet/datasets/coco.py", line 322, in results2json
    json_results = self._det2json(results)
  File "/home/xxxxx/xxxxx/mmdetection/mmdet/datasets/coco.py", line 259, in _det2json
    data['category_id'] = self.cat_ids[label]
IndexError: list index out of range

list out of range here

CASE2: so I change the faster rcnn r50 fpn num_classes to 15 which equal to dataset_a class num at the same time i change the /mmdet/dataset/coco.py CLASSES part to only include dataset_A classes and the /mmdet/dataset/voc.py CLASSES part to only include dataset_B classes

but i still got issue

Done (t=0.94s)
creating index...
Done (t=0.96s)
creating index...
index created!
index created!
2022-09-27 15:40:25,563 - mmcv - INFO - Reducer buckets have been rebuilt in this iteration.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:194: nll_loss_forward_no_reduce_cuda_kernel: block: [1,0,0], thread: [0,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:194: nll_loss_forward_no_reduce_cuda_kernel: block: [1,0,0], thread: [1,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:194: nll_loss_forward_no_reduce_cuda_kernel: block: [1,0,0], thread: [2,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
/home/shuzhilian/anaconda3/envs/openmmlab/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 14 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

CASE3: SO I TRY LIKE THIS WAY and this num_classes still 35

train_class=('plane', 'baseball-diamond',"bridge","ground-track-field", "small-vehicle", "large-vehicle","ship", "tennis-court", 
              "basketball-court","storage-tank","soccer-ball-field","roundabout","harbor","swimming-pool","helicopter",'A1', 'A2', 'A3', 'A4', 
               'A5', 'A6', 'A7',
               'A8', 'A9', 'A10', 'A11', 'A12', 'A13',
               'A14', 'A15', 'A16', 'A17', 'A18', 'A19',
               'A20')
dota_class=('plane', 'baseball-diamond',"bridge","ground-track-field", "small-vehicle", "large-vehicle","ship", "tennis-court", 
              "basketball-court","storage-tank","soccer-ball-field","roundabout","harbor","swimming-pool","helicopter")
mar20_class=('A1', 'A2', 'A3', 'A4', 'A5', 'A6', 'A7',
               'A8', 'A9', 'A10', 'A11', 'A12', 'A13',
               'A14', 'A15', 'A16', 'A17', 'A18', 'A19',
               'A20')
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(type='Resize', img_scale=(1024, 1024), keep_ratio=True),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(type='Normalize', **img_norm_cfg),
    dict(type='Pad', size_divisor=32),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']),
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1024, 1024),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip'),
            dict(type='Normalize', **img_norm_cfg),
            dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img']),
        ])
]
dota_train_dict=dict(
          type='RepeatDataset',
          times=1,
          dataset=dict(
              type='CocoDataset',
              classes=train_class,
              ann_file='ann_path/train_ann.json',
              img_prefix=img_prefix/images/',
              pipeline=train_pipeline
          )
      )
      dota_val_dict=dict(
              type='CocoDataset',
               classes=dota_class,
              ann_file='ann_path/val_ann.json',
              img_prefix=img_prefix/images/',
              pipeline=test_pipeline    
      )
      dota_test_dict=dict(
              type='CocoDataset',
               classes=dota_class,
              ann_file='ann_path/val_ann.json',
              img_prefix=img_prefix/images/',
              pipeline=test_pipeline
           
      )
      mar20_train_dict=dict(
          type='RepeatDataset',
          times=3,
          dataset=dict(
              type='VOCDataset',
               classes=train_class,
              ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
              img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
              pipeline=train_pipeline
          )
      )
      # mar20_val_dict=dict(
      #     dataset=dict(
      #         type='VOCDataset',
      #        ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
      #        img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
      #         pipeline=test_pipeline
      #     )    
      # )
      # mar20_test_dict=dict(
      #     dataset=dict(
      #         type='VOCDataset',
      #        ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
      #        img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
      #         pipeline=test_pipeline
      #     )    
      # )
data = dict(
    samples_per_gpu=4,
    workers_per_gpu=2,
    train=[dota_train_dict,mar20_train_dict],
    val=dota_val_dict,
    test=dota_test_dict
    )
evaluation = dict(interval=1, metric='bbox')

IT SHOWS THE SAME PROBLEM AT CASE1

I really wonder how to correctly concat two different types of datasets for normal training, val and testing processes and another question is : it seem that i can not val and test this two different types datasets together after each epoch finished use the config below：

       mar20_val_dict=dict(
           dataset=dict(
               type='VOCDataset',
              ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
             img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
              pipeline=test_pipeline
           )    
       )
       mar20_test_dict=dict(
           dataset=dict(
               type='VOCDataset',
              ann_file='xxxxxxxxxx/ImageSets/Main/train.txt',
              img_prefix='xxxxxxxxxxxxxxxx/JPEGImage/',
               pipeline=test_pipeline
           )    
       )
      data = dict(
          samples_per_gpu=4,
          workers_per_gpu=2,
          train=[dota_train_dict,mar20_train_dict],
          val=[dota_val_dict,mar20_val_dict]
          test=[dota_test_dict,mar20_test_dict]
          )
      evaluation = dict(interval=1, metric='bbox')
      evaluation = dict(interval=1, metric='mAP')

Environment

fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). sys.platform: linux Python: 3.9.7 (default, Sep 16 2021, 13:09:58) [GCC 7.5.0] CUDA available: True GPU 0,2,3,4,6,7: TITAN RTX GPU 1,5: Tesla P40 CUDA_HOME: :/usr/local/cuda-10.2 GCC: gcc (Ubuntu 7.5.0-3ubuntu1~19.10) 7.5.0 PyTorch: 1.10.0+cu102 PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• LAPACK is enabled (usually provided by MKL)
• NNPACK is enabled
• CPU capability usage: AVX512
• CUDA Runtime 10.2
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
• CuDNN 7.6.5
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.11.1+cu102 OpenCV: 4.5.4 MMCV: 1.4.0 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 10.2 MMDetection: 2.19.0+

Additional information

No response