OVO-bench evaluation prompts are inconsistent to its official repo.
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start by comparing the task prompt definitions in ovo_constants.py with the official evaluation prompt shown in the issue, focusing on the Forward task. Trace where these prompts enter evaluation and determine whether the wording differences affect performance; done means the discrepancy and its impact are explained or documented.
Written by the indexing model from the issue text.
Description
Hi, authors. The findings in your paper are amazing. I find that the prompts in ovo_constants.py is inconsistent to the official eval prompt, especially for the Forward task. Could you explain why you design different prompts? Does this impact the performance?
Official prompts:
# Prompt template for backward-tracing and real-time visual perception task
BR_PROMPT_TEMPLATE = """
Question: {}
Options:
{}
Respond only with the letter corresponding to your chosen option (e.g., A, B, C).
Do not include any additional text or explanation in your response.
"""
# Prompt template for REC task
# REC_PROMPT_TEMPLATE = """
# You're provided with multiple images which are frames extracted from a video, in which the man/woman are performing an action repetitively.
# Now, answer the following question: Have the person in the video {} {} times?
# Answer only with “Yes” or “No”.
# Do not include any additional text or explanation in your response.
# """
REC_PROMPT_TEMPLATE = """
You're watching a video in which people may perform a certain type of action repetively.
The person performing this kind of action are referred to as 'they' in the following statement.
You're task is to count how many times have different people in the video perform this kind of action in total.
One complete motion counts as one.
Now, answer the following question: {}
Provide your answer as a single number (e.g., 0, 1, 2, 3…) indicating the total count.
Do not include any additional text or explanation in your response.
"""
# Prompt template for SSR task
# SSR_PROMPT_TEMPLATE = """
# You're provided with multiple images which are frames extracted from a tutorial video, in which the whole process may contain multiple different steps.
# Now, answer the following question: Have the person in the video complete {}?
# Answer only with “Yes” or “No”.
# Do not include any additional text or explanation in your response.
# """
SSR_PROMPT_TEMPLATE = """
You're watching a tutorial video which contain a sequential of steps.
The following is one step from the whole procedures:
{}
Your task is to determine if the man or woman in the video is currently performing this step.
Answer only with “Yes” or “No”.
Do not include any additional text or explanation in your response.
"""
# Prompt template for CRR task
CRR_PROMPT_TEMPLATE = """
You're responsible of answering questions based on the video content.
The following question are relevant to the latest frames, i.e. the end of the video.
{}
Decide whether existing visual content, especially latest frames, i.e. frames that near the end of the video, provide enough information for answering the question.
Answer only with “Yes” or “No”.
Do not include any additional text or explanation in your response.
"""
- Dominant language
- Python
- Stars
- 167
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from EvolvingLMMs-Lab/SimpleStream
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
All issues in EvolvingLMMs-Lab/SimpleStream
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100