Failed to reproduce mova-360p’s LSE-C and LSE-D metrics on Verse-Bench
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- huggingface, python
- Domain
- machine-learning, testing-qa
Research direction
Start with the Verse-Bench Hugging Face dataset, GitHub repository, and the office code demo settings used for video inference. Compare the preparation, inference, and LSE-C/LSE-D evaluation steps against the reported paper results and the uploaded set3 cases; done means identifying the source of the discrepancy or explaining why the reported metrics cannot be reproduced.
Written by the indexing model from the issue text.
Description
Hi,
I tried to reproduce mova-360p’s LSE-C and LSE-D metrics on Verse-Bench, but ran into some issues. I followed the instructions in Verse-Bench (Hugging Face dataset) and Verse-Bench (GitHub repo) to prepare the test data and run the evaluation. For video inference, I used the same settings as the office code demo.
However, my average LSE-C is 5.41 and the average LSE-D is 13.303, which are much worse than the numbers reported in the paper. When I visualized the generated videos, they didn’t look obviously strange.
I uploaded the first two inference results from set3 here:
Google Drive link
In my tests:
- For case0: LSE-C = 4.4768114, LSE-D = 12.80221240751205
- For case-1: LSE-C = 6.9611187, LSE-D = 13.244998070501513
I’m not sure what might be causing this discrepancy.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 97
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from OpenMOSS/MOVA
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 3/5 1-2 days Newbie friendliness 48/100
-
Difficulty 4/5 3-5 days Newbie friendliness 38/100
-
Difficulty 3/5 1-2 days Newbie friendliness 48/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
Similar issues
-
comp/tools P2 type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NousResearch/hermes-agent#142908 · 1 comment ·
-
[oblt-aw][autodoc] Fix malformed headings in new Buildkite docs pagesPossibly taken @github-actions claimed this today. Opendocumentation
Difficulty 1/5 Under an hour Newbie friendliness 83/100
Maintainers usually reply within 1 day
-
[Bug]: fileUploadMatrix.test.ts URL case does a real DNS lookup and fails offlinePossibly taken @r2jaff claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
ComposioHQ/composio#5054 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
pytorch/tensordict#3286 ·
Maintainers usually reply within 1 day