Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Request] Release optimized skill artifacts for additional methods / models / harnesses

Open
  • Is something wrong?
#130 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 4 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
release

Research direction

The currently released artifacts are in ckpt/ as compact best_skill.md files; start by reviewing those files and the Table 1 cells listed in the issue. Done means publishing the requested additional artifacts and, where available, their accepted-edit histories and selection/test scores.

Written by the indexing model from the issue text.

Description

Hi SkillOpt team,

Thank you for open-sourcing the best GPT-5.5 SkillOpt checkpoints under ckpt/
(https://github.com/microsoft/SkillOpt/tree/main/ckpt) — they have been very
helpful for getting started.

Context

I'm doing an analysis of the learned skills and would like to build on SkillOpt.
Since optimizing these skills is token-expensive and I'm compute/budget-constrained,
re-running the full loop to reproduce every cell in Table 1 is impractical on my end.
Because the exported artifacts are compact best_skill.md files (per the paper, mostly
<2,000 tokens), releasing them should be low-cost on your side and would let the
community study and compare the skills directly, and improve reproducibility.

Request

Would it be possible to release the exported skill artifacts (the final best_skill.md,
and ideally the accepted-edit history / selection & test scores if available) for the
following cells?

1. Direct-chat artifacts — one skill per benchmark
(SearchQA, SpreadsheetBench, OfficeQA, DocVQA, LiveMath, ALFWorld):

Method GPT-5.5 GPT-5.4 GPT-5.4-mini GPT-5.4-nano GPT-5.2 Qwen3.5-4B Qwen3.6-35B-A3B
SkillOpt ✅ done ⬜ ⬜ ⬜ ⬜ ⬜ ⬜
Trace2Skill ⬜ ⬜ ⬜ ⬜ ⬜ ⬜ ⬜
GEPA ⬜ ⬜ ⬜ ⬜ ⬜ ⬜ ⬜

2. Harness artifacts (GPT-5.5) — 5 benchmarks, no ALFWorld:

Method Codex harness Claude Code harness
SkillOpt ⬜ ⬜
EvoSkill ⬜ ⬜

Note: I limited the EvoSkill request to the Codex / Claude Code harnesses because,
if I read Table 1 correctly, EvoSkill is only reported there and not in the
direct-chat rows. Please correct me if direct-chat EvoSkill artifacts also exist.

Priority (in case a full release is too much at once)

If releasing everything is impractical, an incremental release in this order would
already be very useful to me:

  1. SkillOpt across the remaining 6 direct-chat models (the headline artifacts).
  2. GEPA and Trace2Skill across all 7 direct-chat models (for cross-method comparison).
  3. The Codex / Claude Code harness artifacts (SkillOpt + EvoSkill).

Happy to help however useful — e.g. verifying/organizing the released files, or writing
a short README for the extra checkpoints — and I will of course cite the paper.

Thanks again for the great work and for open-sourcing it!

Dominant language
Python
Stars
17.3k
Forks
1.6k
Avg merge
8d 16h
Merged PRs (30d)
11

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/SkillOpt

All issues in microsoft/SkillOpt

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.