Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

CharXiv reasoning selects instructions from question source instead of answer type (766/1000 validation prompts)

Open Beginner friendly
  • Is something wrong?
#27 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
Half a day
Newbie friendliness
62/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
machine-learning

Research direction

The bug is in the field lookup in lmms_eval_tasks/charxiv/utils.py, where charxiv_reasoning_doc_to_text_cot picks the instruction from reasoning_q_source instead of reasoning_a_type. Start by reproducing the report's example doc, then switch the selection to reasoning_a_type. Done when all four answer types select their matching instruction, including the category-4 integer and decimal cases, and regression tests cover them. Check the linked upstream lmms-eval change before deciding how to handle historical scores.

Written by the indexing model from the issue text.

Description

Summary

CharXiv reasoning uses reasoning_q_source to select the answer instruction, but that field records who wrote the question, not its answer type. In the public Hugging Face dataset, reasoning_a_type already corresponds to official CharXiv's inst_category.

Verified at 08d755b973e4ad990ac5cdd64fb992c804d840db: the field lookup.

Reproduction

From the repository's evaluation environment:

from lmms_eval_tasks.charxiv.utils import charxiv_reasoning_doc_to_text_cot

doc = {
    "figure_path": "images/2.jpg",
    "reasoning_q": "What is the name of the line that is the furthest away from its fi value from the W_H axis?",
    "reasoning_a": "lambda_L = 0.13",
    "reasoning_a_type": 1,  # text in chart
    "reasoning_q_source": 3,  # human-written
}
print(charxiv_reasoning_doc_to_text_cot(doc))

Actual instruction: Your final answer must be grounded to a number ...

Expected instruction: Your final answer must be grounded to some text ..., using answer type 1.

Evidence and impact

  • The official record for figure 2 has inst_category=1, qa_source=3. The official prompt builder selects by inst_category.
  • On all 1,000 public validation records at HF revision f441eb632fc62f6f777830a0f47619e6e86459b0, reasoning_a_type agrees with official inst_category on 1,000/1,000; question source agrees on only 234/1,000. Executing this repository's prompt-builder definitions and constants gives 766 changed prompts when selecting the actual answer type.
  • All 229 number-in-general questions have answer type 4, but source only takes values 1–3, so the current lookup never selects their integer/decimal-precision instruction.

Suggested correction: inst_category = doc["reasoning_a_type"]. Cover all four answer types, including integer and decimal category-4 answers, with regression tests. Historical scores would need to identify which prompt convention was used.

This report concerns prompt selection; the reward function reads reasoning_a separately. Verification here was offline source/data comparison, without model inference or judge calls in LongVT. The same prompt-selection defect is addressed upstream in lmms-eval #1583.

Dominant language
Python
Stars
270
Forks
17
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from EvolvingLMMs-Lab/LongVT

All issues in EvolvingLMMs-Lab/LongVT

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.