AttributeError: `EasyDict` object has no attribute `moment_layers` when loading pre-trained CG-DETR checkpoints.

まず lighthouse/models.py のチェックポイント読み込みを追跡し、次に lighthouse/common/cg_detr.py と lighthouse/common/cg_detr_transformer.py のモデル構築パスを、保存されたチェックポイントのオプションと比較します。現在のコードで必要とされている設定キーのうち、提供された重みに存在しないものを確認します。README にリンクされた CG-DETR チェックポイントが、属性不足エラーなしで CGDETRPredictor 経由で読み込まれれば完了です。

Loading a pre-trained CG-DETR checkpoint from the uploaded pre-trained weights fails with the following error:

AttributeError: 'EasyDict' object has no attribute 'moment_layers'

To reproduce
Download pre-trained model weights from the google drive URL mentioned in the Readme and then load a CG-DETR checkpoint using lighthouse.models.CGDETRPredictor.

from lighthouse.models import CGDETRPredictor
model = CGDETRPredictor('results/cg_detr/qvhighlight/clip/best.ckpt', device='cpu', feature_name='clip')

Full traceback

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[19], line 1
----> 1 model = CGDETRPredictor('results/cg_detr/qvhighlight/clip/best.ckpt', device='cpu',
      2                         feature_name='clip')

File .../.venv/lib/python3.9/site-packages/lighthouse/models.py:363, in CGDETRPredictor.__init__(self, ckpt_path, device, feature_name, slowfast_path, pann_path)
    355 def __init__(
    356     self, 
    357     ckpt_path: str,
   (...)
    361     pann_path: Optional[str] = None
    362     ) -> None:
--> 363     super().__init__('cg_detr', ckpt_path, device, 
    364                      feature_name, slowfast_path, pann_path)

File .../.venv/lib/python3.9/site-packages/lighthouse/models.py:85, in BasePredictor.__init__(self, model_name, ckpt_path, device, feature_name, slowfast_path, pann_path)
     81     self._audio_encoder = self._initialize_audio_encoder(feature_name, pann_path)
     83 self._text_encoder: TextEncoder = self._initialize_text_encoder(feature_name)
---> 85 self._model: torch.nn.Module = self._initialize_model(args, model_name)
     86 self._load_weights(ckpt['model'])
     88 self._feature_name: str = feature_name

File .../.venv/lib/python3.9/site-packages/lighthouse/models.py:107, in BasePredictor._initialize_model(self, args, model_name)
     96 model_builders = {
     97     'moment_detr': build_model_moment_detr,
     98     'qd_detr': build_model_qd_detr,
   (...)
    103     'taskweave': build_model_task_weave
    104 }
    106 if model_name in model_builders:
--> 107     model, _ = model_builders[model_name](args)
    108     return model
    109 else:

File .../.venv/lib/python3.9/site-packages/lighthouse/common/cg_detr.py:998, in build_model(args)
    995 def build_model(args):
    996     device = torch.device(args.device)
--> 998     transformer = build_transformer(args)
    999     position_embedding, txt_position_embedding = build_position_encoding(args)
   1001     model = CGDETR(
   1002         transformer,
   1003         position_embedding,
   (...)
   1013         args=args
   1014     )

File .../.venv/lib/python3.9/site-packages/lighthouse/common/cg_detr_transformer.py:883, in build_transformer(args)
    882 def build_transformer(args):
--> 883     return Transformer(
    884         d_model=args.hidden_dim,
    885         dropout=args.dropout,
    886         nhead=args.nheads,
    887         dim_feedforward=args.dim_feedforward,
    888         num_encoder_layers=args.enc_layers,
    889         num_decoder_layers=args.dec_layers,
    890         normalize_before=False,
    891         return_intermediate_dec=True,
    892         activation='prelu',
    893         args=args
    894     )

File .../.venv/lib/python3.9/site-packages/lighthouse/common/cg_detr_transformer.py:150, in Transformer.__init__(self, d_model, nhead, num_queries, num_encoder_layers, num_decoder_layers, dim_feedforward, dropout, activation, normalize_before, return_intermediate_dec, query_dim, keep_query_pos, query_scale_type, num_patterns, modulate_t_attn, bbox_embed_diff_each_layer, args)
    147 mcls_encoder_layer = TransformerEncoderLayer(d_model, nhead, dim_feedforward,
    148                                         dropout, activation, normalize_before)
    149 mcls_encoder_norm = nn.LayerNorm(d_model) if normalize_before else None
--> 150 self.mcls_encoder = TransformerEncoder(mcls_encoder_layer, args.moment_layers, mcls_encoder_norm)
    152 t2v_encoder_layer = T2V_TransformerEncoderLayer(d_model, nhead, dim_feedforward,
    153                                         dropout, activation, normalize_before, self.args.num_dummies)
    154 encoder_norm = nn.LayerNorm(d_model) if normalize_before else None

AttributeError: 'EasyDict' object has no attribute 'moment_layers'

The checkpoint doesn't contain the moment_layers key although the code expects it to. Maybe the checkpoint was generated on an older version where this attribute wasn't saved?

I attempted to work around this by patching the checkpoint to include moment_layers:

if not hasattr(ckpt['opt'], 'moment_layers'):
    ckpt['opt']['moment_layers'] = 1  # Default value for moment_layers

torch.save(ckpt, 'results/cg_detr/qvhighlight/clip_slowfast/best_patched.ckpt')

Loading the model with the patched checkpoint gave a new error: AttributeError: 'EasyDict' object has no attribute 'num_dummies.

It seems like these checkpoints are outdated/incompatible with the current Lighthouse version and should either be replaced by new ones or the code made compatible with these checkpoints. I might've just misunderstood the inference instructions though and I'm doing something wrong on my end, in which case sorry for the issue!