Not able to reproduce the stage two models

提供された再現手順を demo/example1.mp4 で実行し、その後 demo/internvideo2_stage2_config.py、demo_config.Config.from_file、demo.utils.setup_internvideo2 を調査してください。第2段階のチェックポイント、重み、語彙サイズがどのように読み込まれるかを確認し、報告された上位5つの確率を比較してください。読み込みが不十分な転移結果の原因となっているかを特定し、再現可能な調査結果または確認済みの修正を文書化できれば完了です。

during model loading the check points weight and vocab size seems to be wrong
below is the code I used to generate this result, which has also been mentioned by others, I also tried clip and other models, the reuslts seems to be pretty bad when transferring to other dataset

text: A man in a gray sweater plays fetch with his dog in the snowy yard, throwing a toy and watching it run. ~ prob: 0.6796
text: A man in a gray hat and coat walks through the snowy yard, carefully navigating around the trees. ~ prob: 0.0944
text: A person dressed in a blue jacket shovels the snow-covered pavement outside their house. ~ prob: 0.0754
text: A person stands on the snowy floor, pushing a sled loaded with blankets, preparing for a fun-filled ride. ~ prob: 0.0375
text: A playful dog slides down a snowy hill, wagging its tail with delight. ~ prob: 0.0288

Looking for your guidance.

`import numpy as np
import os
import io
import cv2
os.environ['CUDA_LAUNCH_BLOCKING']='1'
import torch

from demo_config import (Config,
eval_dict_leaf)

from demo.utils import (retrieve_text,
_frame_from_video,
setup_internvideo2)
seed = 4491734
print("Seed:", seed)

np.random.seed(seed)
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(seed)
video = cv2.VideoCapture('demo/example1.mp4')
frames = [x for x in _frame_from_video(video)]
text_candidates = ["A playful dog and its owner wrestle in the snowy yard, chasing each other with joyous abandon.",
"A man in a gray coat walks through the snowy landscape, pulling a sleigh loaded with toys.",
"A person dressed in a blue jacket shovels the snow-covered pavement outside their house.",
"A pet dog excitedly runs through the snowy yard, chasing a toy thrown by its owner.",
"A person stands on the snowy floor, pushing a sled loaded with blankets, preparing for a fun-filled ride.",
"A man in a gray hat and coat walks through the snowy yard, carefully navigating around the trees.",
"A playful dog slides down a snowy hill, wagging its tail with delight.",
"A person in a blue jacket walks their pet on a leash, enjoying a peaceful winter walk among the trees.",
"A man in a gray sweater plays fetch with his dog in the snowy yard, throwing a toy and watching it run.",
"A person bundled up in a blanket walks through the snowy landscape, enjoying the serene winter scenery."]
#%%
config = Config.from_file('demo/internvideo2_stage2_config.py')
config = eval_dict_leaf(config)
#%%

config['pretrained_path'] = '/InternVideo/InternVideo2/multi_modality/weights/InternVideo2-stage2_1b-224p-f4.pt',

intern_model, tokenizer = setup_internvideo2(config)
#%%
texts, probs = retrieve_text(frames, text_candidates, model=intern_model.eval(), topk=5, config=config)

for t, p in zip(texts, probs):
print(f'text: {t} ~ prob: {p:.4f}')`