Uniform predictions (identical saliency scores) during TVSum dataset evaluation with Lighthouse pipeline
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- computer-vision, machine-learning
Research direction
Start with training/evaluate.py and reproduce the issue using the provided TVSum evaluation command, checkpoint, and JSONL path. Inspect the loaded i3d_clip feature shapes, data loading, and post-processing while comparing the predicted scores across time windows. Done means identifying the cause of the identical predictions and confirming that evaluation produces non-uniform scores.
Written by the indexing model from the issue text.
Description
When evaluating the Lighthouse model on the TVSum dataset using the provided scripts, the model outputs identical highlight scores across all time windows, despite following the tutorial steps.
eval_command:
python /lighthouse/training/evaluate.py --model uvcom --dataset tvsum --feature i3d_clip --domain BK --split val --model_path checkpoints/results/uvcom/tvsum/i3d_clip/BK/best.ckpt --eval_path data/tvsum/tvsum_val_release.jsonl
I printed out the saliency score predictions during the evaluation, and a screenshot is attached below to show the output.
1.Are there known issues with TVSum evaluation in the current version?
2.Could the problem stem from data loading or post-processing steps or shape problem of loaded features?
However, i downloaded all the checkpoints and extracted-features from the link on lighthouse repo page.
Appreciate your guidance to resolve this issue.
- Dominant language
- Python
- Stars
- 272
- Forks
- 22
- Avg merge
- 18h 54m
- Merged PRs (30d)
- 1
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from line/lighthouse
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
line/lighthouse#75 · 1 comment ·
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
line/lighthouse#74 · 2 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
line/lighthouse#72 · 1 comment ·
-
Support UMTOpen
Difficulty 5/5 Over a week Newbie friendliness 20/100
line/lighthouse#60 ·
-
Support R2-TuningOpen
Difficulty 5/5 Over a week Newbie friendliness 25/100
line/lighthouse#59 ·
Similar issues
-
Wayback LinksOpen
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
chnm/teachinghistory-hugo#53 ·
-
gpt-oss "Forcing a tool call" row no longer reproduces: tool_choice is now honoured on every surfaceOpen
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
aws-samples/sample-per-model-bedrock#840 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
521xueweihan/HelloGitHub#4318 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
DiamondLightSource/eiger-correction#2 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
django-cms/djangocms-frontend#413 ·