AttributeError: `EasyDict` object has no attribute `moment_layers` when loading pre-trained CG-DETR checkpoints.

Bắt đầu bằng cách theo dõi việc tải checkpoint trong lighthouse/models.py, sau đó so sánh các luồng khởi tạo model trong lighthouse/common/cg_detr.py và lighthouse/common/cg_detr_transformer.py với các tùy chọn của checkpoint đã lưu. Kiểm tra những khóa cấu hình nào được code hiện tại yêu cầu nhưng không có trong các trọng số được cung cấp. Công việc được xem là hoàn tất khi checkpoint CG-DETR được liên kết trong README tải được thông qua CGDETRPredictor mà không có lỗi thiếu thuộc tính.

Loading a pre-trained CG-DETR checkpoint from the uploaded pre-trained weights fails with the following error:

AttributeError: 'EasyDict' object has no attribute 'moment_layers'

To reproduce
Download pre-trained model weights from the google drive URL mentioned in the Readme and then load a CG-DETR checkpoint using lighthouse.models.CGDETRPredictor.

from lighthouse.models import CGDETRPredictor
model = CGDETRPredictor('results/cg_detr/qvhighlight/clip/best.ckpt', device='cpu', feature_name='clip')

Full traceback

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[19], line 1
----> 1 model = CGDETRPredictor('results/cg_detr/qvhighlight/clip/best.ckpt', device='cpu',
      2                         feature_name='clip')

File .../.venv/lib/python3.9/site-packages/lighthouse/models.py:363, in CGDETRPredictor.__init__(self, ckpt_path, device, feature_name, slowfast_path, pann_path)
    355 def __init__(
    356     self, 
    357     ckpt_path: str,
   (...)
    361     pann_path: Optional[str] = None
    362     ) -> None:
--> 363     super().__init__('cg_detr', ckpt_path, device, 
    364                      feature_name, slowfast_path, pann_path)

File .../.venv/lib/python3.9/site-packages/lighthouse/models.py:85, in BasePredictor.__init__(self, model_name, ckpt_path, device, feature_name, slowfast_path, pann_path)
     81     self._audio_encoder = self._initialize_audio_encoder(feature_name, pann_path)
     83 self._text_encoder: TextEncoder = self._initialize_text_encoder(feature_name)
---> 85 self._model: torch.nn.Module = self._initialize_model(args, model_name)
     86 self._load_weights(ckpt['model'])
     88 self._feature_name: str = feature_name

File .../.venv/lib/python3.9/site-packages/lighthouse/models.py:107, in BasePredictor._initialize_model(self, args, model_name)
     96 model_builders = {
     97     'moment_detr': build_model_moment_detr,
     98     'qd_detr': build_model_qd_detr,
   (...)
    103     'taskweave': build_model_task_weave
    104 }
    106 if model_name in model_builders:
--> 107     model, _ = model_builders[model_name](args)
    108     return model
    109 else:

File .../.venv/lib/python3.9/site-packages/lighthouse/common/cg_detr.py:998, in build_model(args)
    995 def build_model(args):
    996     device = torch.device(args.device)
--> 998     transformer = build_transformer(args)
    999     position_embedding, txt_position_embedding = build_position_encoding(args)
   1001     model = CGDETR(
   1002         transformer,
   1003         position_embedding,
   (...)
   1013         args=args
   1014     )

File .../.venv/lib/python3.9/site-packages/lighthouse/common/cg_detr_transformer.py:883, in build_transformer(args)
    882 def build_transformer(args):
--> 883     return Transformer(
    884         d_model=args.hidden_dim,
    885         dropout=args.dropout,
    886         nhead=args.nheads,
    887         dim_feedforward=args.dim_feedforward,
    888         num_encoder_layers=args.enc_layers,
    889         num_decoder_layers=args.dec_layers,
    890         normalize_before=False,
    891         return_intermediate_dec=True,
    892         activation='prelu',
    893         args=args
    894     )

File .../.venv/lib/python3.9/site-packages/lighthouse/common/cg_detr_transformer.py:150, in Transformer.__init__(self, d_model, nhead, num_queries, num_encoder_layers, num_decoder_layers, dim_feedforward, dropout, activation, normalize_before, return_intermediate_dec, query_dim, keep_query_pos, query_scale_type, num_patterns, modulate_t_attn, bbox_embed_diff_each_layer, args)
    147 mcls_encoder_layer = TransformerEncoderLayer(d_model, nhead, dim_feedforward,
    148                                         dropout, activation, normalize_before)
    149 mcls_encoder_norm = nn.LayerNorm(d_model) if normalize_before else None
--> 150 self.mcls_encoder = TransformerEncoder(mcls_encoder_layer, args.moment_layers, mcls_encoder_norm)
    152 t2v_encoder_layer = T2V_TransformerEncoderLayer(d_model, nhead, dim_feedforward,
    153                                         dropout, activation, normalize_before, self.args.num_dummies)
    154 encoder_norm = nn.LayerNorm(d_model) if normalize_before else None

AttributeError: 'EasyDict' object has no attribute 'moment_layers'

The checkpoint doesn't contain the moment_layers key although the code expects it to. Maybe the checkpoint was generated on an older version where this attribute wasn't saved?

I attempted to work around this by patching the checkpoint to include moment_layers:

if not hasattr(ckpt['opt'], 'moment_layers'):
    ckpt['opt']['moment_layers'] = 1  # Default value for moment_layers

torch.save(ckpt, 'results/cg_detr/qvhighlight/clip_slowfast/best_patched.ckpt')

Loading the model with the patched checkpoint gave a new error: AttributeError: 'EasyDict' object has no attribute 'num_dummies.

It seems like these checkpoints are outdated/incompatible with the current Lighthouse version and should either be replaced by new ones or the code made compatible with these checkpoints. I might've just misunderstood the inference instructions though and I'm doing something wrong on my end, in which case sorry for the issue!