Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Mining is limited to programmatic success checks — intent-level improvements get dropped or Goodharted into shallow proxy rules

Open
  • Is something wrong?
#154 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 4 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
python
Domain
ai

Research direction

Start by tracing the task miner's regex-based success checks and the mining/replay flow described in the issue. Compare the proposed divergence-point, clean-context replay, and baseline-instance approaches, then review the maintainers' response to determine whether a judge-based evidence standard is accepted; the issue is complete only when a concrete design and acceptance criteria are agreed.

Written by the indexing model from the issue text.

Description

ORIGIN & MOTIVATION

I'm exploring using SkillOpt as a step toward "on-the-job training" for AI Employees learning a specific job role from real work sessions. This issue is about what looks like a structural ceiling on that goal in the mining stage. (Related but distinct from #67, which covers reward-hacking on the gate side; this is about the miner side.)

Problem

The task miner only retains a candidate task if success can be expressed as a programmatic check (regex-style assertions). That means behaviors like "understand my intent better" don't compress into a check, so they are either:

  1. dropped the most valuable improvement signal never enters the pipeline, or
  2. mangled into a shallow proxy a phrasing/format preference observed in one session gets distilled into a hyper-literal rule that is then robotically applied to all future outputs. Goodhart's law in miniature: the check becomes the target.

I've observed the second failure mode in my own runs: a one-off summary phrasing preference became a rigid rule stamped onto every future summary.

If mining is restricted to regex-expressible outcomes, the system can only ever improve the regex-expressible slice of agent behavior, which is a small subset of what users actually repeat-and-rephrase about in real sessions.

Possible directions (heuristics, not designs)

  1. Divergence-point detection: mine topic-shift structure in transcripts (user descends into a rabbit hole on a sub-issue → resolves it → conversation returns to the main thread) as natural task boundaries and implicit failure signals, instead of requiring a programmatic outcome check.
  2. Clean-context replay comparison: a second instance with fresh context attempts the reconstructed task; a judge compares its output against what the user ultimately accepted in the original transcript, rather than against a regex proxy.
  3. Baseline-instance divergence: an instance seeded only with the user's global rules/skills replays the session's user prompts in order until its output clearly and definitely diverges from the transcript — the divergence point marks where a learnable behavior lives, and bounds the task to mine.

Question for maintainers

Is there interest in supporting judge-based (non-programmatic) success checks in mining/replay, and what evidence standard would you consider acceptable for gating on them?

Dominant language
Python
Stars
18k
Forks
1.7k
Avg merge
8d 16h
Merged PRs (30d)
11

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/SkillOpt

All issues in microsoft/SkillOpt

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.