okl metric: stale-use and over-scope rates

Open
  • Is something wrong?
#252 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
47/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
ai-infra-agents, developer-experience

Research direction

Start by reading the okl metric entry point, the briefing log code that records per-lesson drift, and the store's handling of repo: scopes. Then inspect #189's tests and the seed store under evals/; done means both rates include denominators, over-scope lessons are listed, and the baseline is recorded.

Written by the indexing model from the issue text.

Description

Why

DreamBench-SWE (arXiv 2608.20664, August 2026) grades software-agent memory with executable hygiene diagnostics rather than a single score, and reports them for its reference system: StaleUseRate 0.200, OverscopeRate 0.167, RepeatedErrorRate 0.571. Two of those have direct okl equivalents that nothing reports today. okl metric reports recurrence and exposure only.

The two rates

  • Stale-use rate: the share of briefings that included a lesson whose drift was red at the time. The briefing already annotates drift per lesson, so the data exists at briefing time; the metric needs the briefing log to keep that flag and the metric to count it. A lesson can be stale and still correct, so the rate is a hygiene signal, not a defect count; #231 covers the related TTL gap and #222 the proven-versus-unwatched distinction.
  • Over-scope rate: the share of org-scope lessons whose text names one repository's paths or project name. A heuristic will do to start: path-like tokens, and the repo names the store already knows from its repo: scopes. Report it from okl metric, with the offending lessons listed so each can be re-scoped or generalised.

Done when

okl metric prints both rates beside recurrence and exposure, with the denominator for each; the seed store's rates are recorded once as a baseline in evals/; and #189's tests for wrong and stale lessons can assert on them.

🤖 Generated with Claude Code

Dominant language
Python
Stars
1
Forks
1
Avg merge
5h 50m
Merged PRs (30d)
125

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from emeraldleaf/okl

All issues in emeraldleaf/okl

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.