SHAPER VLABench reproduction: 800-episode evaluation split and sandbox configuration
Maintainers usually reply within 2 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 48/100
- Issue type
- Documentation
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- machine-learning, tooling
Research direction
Start with the linked SHAPER recipe and inspect its episode configuration plus harness_optimizer.txt to identify the defined training and validation splits and sandbox modules. Trace the evaluation entry point and documentation for the command that generates or consumes the 800-episode C1–C4 manifests. Reproduction is complete when the exact manifests, runtime configuration, command, and final evolved skill/harness artifacts are identified and verified.
Written by the indexing model from the issue text.
Description
Hi, I’m trying to reproduce the VLABench results using the SHAPER recipe on v0.x.
I have two questions:
- The recipe defines 15 training and 24 validation episodes, while the paper reports 800 held-out episodes across C1–C4. Where can I find the exact evaluation manifests and the command used to reproduce Table 2?
harness_optimizer.txtsays modules such ascollectionsare preloaded, but the sandbox namespace does not appear to provide them. Generated candidates encounterNameError: name 'collections' is not defined. What runtime configuration should be used?
Could you also share the final evolved skill and harness used for the reported results?
Thank you!
- Dominant language
- Python
- Stars
- 18.8k
- Forks
- 1.7k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 10
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/agent-lightning
-
Low-precision rollout normalization can overflow its divisor and erase advantagesPossibly taken @lll-hhh claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
microsoft/agent-lightning#661 ·
Maintainers usually reply within 2 days
-
RayPPOTrainer._dump_generations() missing gts argument in _train_step callPossibly taken @yunkai-yyk claimed this 254 days ago. Openbug v0.x verl
Difficulty 1/5 Under an hour Newbie friendliness 68/100
microsoft/agent-lightning#492 ·
Maintainers usually reply within 2 days
-
enhancement store v0.x
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
microsoft/agent-lightning#372 · 10 comments ·
Maintainers usually reply within 2 days
-
Terminal rollout projection drops numeric dataset ID zeroPossibly taken @notoufa claimed this 4 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
microsoft/agent-lightning#659 ·
Maintainers usually reply within 2 days
-
Gateway loses model-request events for malformed or non-object upstream JSONPossibly taken @JianhaoYu-AI claimed this 5 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 20/100
microsoft/agent-lightning#655 ·
Maintainers usually reply within 2 days
All issues in microsoft/agent-lightning
Similar issues
-
[oblt-aw][autodoc] Document new required `shared-org-key` input for `obs-aw-dependency-review`Possibly taken @github-actions claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 90/100
Maintainers usually reply within 1 day
-
getchaintips leaves out the active tip once a header is staged on it, where Core always lists itOpen
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
btclib-org/btclib-node#2158 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
dbt-msft/dbt-sqlserver#897 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Gathin23/cleanyourtv#2770 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Dao-AILab/flash-attention#3097 ·
Maintainers usually reply within 3 days