Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

LiveMath scores imply 125 test items, but the released split has 124

Closed
  • Is something wrong?
#157 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 4 days

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
data, documentation

Research direction

Start by comparing the released LiveMath manifest counts with the denominators implied by Table 1, then review the discussion in related issue #108. Done means confirming which split produced the reported scores and releasing the original manifest or per-item results if they are available.

Written by the indexing model from the issue text.

Description

Hi SkillOpt team,

I found a possible mismatch between the LiveMath split and the Qwen results in Table 1.

The released manifest contains 35/18/124 train/val/test items. However, several reported scores exactly match a denominator of 125:

  • Qwen3.5-4B: No skill 22.4 = 28/125, Trace2Skill 27.2 = 34/125, SkillOpt 52.0 = 65/125
  • Qwen3.6-35B-A3B: No skill 31.2 = 39/125, Trace2Skill 29.6 = 37/125, SkillOpt 41.6 = 52/125

With 124 binary exact-match examples, neither 52.0% nor 41.6% is attainable after rounding.

For 177 total items, a 2:1:7 split can be either:

  • 35/17/125 if train and val are floored and the remainder goes to test;
  • 35/18/124 with the currently released largest-remainder split.

Could you confirm whether Table 1 used the former 35/17/125 split, and release the original LiveMath manifest or per-item results?

Related: #108.

Thanks!

Dominant language
Python
Stars
17.3k
Forks
1.6k
Avg merge
8d 16h
Merged PRs (30d)
11

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/SkillOpt

All issues in microsoft/SkillOpt

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.