[Bug] LiveMath default config expects 2-1-7 split, but downloader and run.sh use 2-2-6
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 75/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Domain
- data, machine-learning
Research direction
Compare configs/livemathematicianbench/default.yaml with the split path in data/download.py and copilot_example/livemath/run.sh. Start by following the documented download workflow, then run the LiveMath training example with the default config. Done means the default config points to the directory created by the downloader and used by the runner.
Written by the indexing model from the issue text.
Description
Description
The default LiveMathematicianBench training configuration points to a split directory that is inconsistent with both the official dataset downloader and the LiveMath example runner.
The config expects:
data/ablation_splits/livemathematicianbench/2-1-7_seed42
However, the official downloader creates, and the LiveMath runner uses:
data/ablation_splits/livemathematicianbench/2-2-6_seed42
As a result, following the documented data-download workflow and then running training with the default LiveMath config fails because the configured directory does not exist.
Relevant files
The default config uses 2-1-7_seed42:
env:
split_mode: split_dir
split_dir: data/ablation_splits/livemathematicianbench/2-1-7_seed42
The official downloader creates 2-2-6_seed42:
"livemath": (
"ablation_splits/livemathematicianbench/2-2-6_seed42",
{"train": "items.json", "val": "items.json", "test": "items.json"},
),
The LiveMath example runner also uses 2-2-6_seed42:
SPLIT_DIR="${PROJECT_ROOT}/data/ablation_splits/livemathematicianbench/2-2-6_seed42"
- Dominant language
- Python
- Stars
- 187
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from EvolvingLMMs-Lab/SkillOpt-Lite
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
EvolvingLMMs-Lab/SkillOpt-Lite#6 · 3 reactions ·
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
EvolvingLMMs-Lab/SkillOpt-Lite#4 · 2 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
EvolvingLMMs-Lab/SkillOpt-Lite#3 · 1 comment ·
-
Security and evaluation integrity: no documented isolation boundary for the optimizer coding agent Open
Difficulty 5/5 Over a week Newbie friendliness 35/100
EvolvingLMMs-Lab/SkillOpt-Lite#2 · 1 comment · 1 reaction ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
EvolvingLMMs-Lab/SkillOpt-Lite#1 · 2 comments ·
All issues in EvolvingLMMs-Lab/SkillOpt-Lite
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100