Experiment: the common-word floor for one-word tasks

Open
  • Is something wrong?
#282 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
20/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
ai-infra-agents

Research direction

Read core._apply_floor, the probe implementation, and test_a_one_word_task_keeps_the_old_rule_even_for_a_common_word; the issue names these entry points but gives no file paths. First inspect how the existing probe builds prompts and measures recall, then add the pre-registered prompts and compare all three arms at one commit against one store. Done means reporting common-word counts, rare-prompt recall, and whether any existing probe row moves.

Written by the indexing model from the issue text.

Description

enhancement

Part of #258; follows #277 and REPORT §4v. Raised in review on #280.

Where it stands (2026-10-24, the common-word-floor branch)

§4v weighs the floor's one-word match by how common the word is: inside a task of several subject words, a single shared word in a record's title or symptom admits it only when at most 5% of the store's live records carry that word (and more than five do). A task of one subject word is untouched: the floor drops to one shared word, so okl check --task "doc" still admits every record that contains "doc" anywhere, common or not. core._apply_floor says so in a comment, and test_a_one_word_task_keeps_the_old_rule_even_for_a_common_word pins it.

It was left alone because nothing measures it: of the probe's 62 prompts (8 authored, 24 of its own rephrasings, 24 held-out, 6 no-match), none has exactly one subject word after briefing_terms.

The question

Should a one-word task of a common word ("doc", "update", "tests") brief nothing, brief only records whose title or symptom holds the word, or keep today's behaviour? A rare one-word task ("pkce", "RUF012") should keep admitting either way; #263 asks the related question for rare identifier-like words in a record's body.

Pre-registration (written before any run)

  • Instrument first: add one-word prompts to the probe: six common words drawn from the store's most frequent title words, six rare identifiers each with a governing lesson. Commit them before measuring.
  • Arms: today's rule; the common-word weighting applied to one-word tasks with the title-or-symptom requirement; the weighting applied with no title requirement.
  • Measure: the six common prompts' counts, the six rare prompts' recall, and every existing row of the probe (held-out, authored, no-match), at one commit against one store.
  • Falsifiers: any rare one-word prompt loses its lesson; any existing row moves.

🤖 Generated with Claude Code

Dominant language
Python
Stars
1
Forks
1
Avg merge
5h 17m
Merged PRs (30d)
128

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from emeraldleaf/okl

All issues in emeraldleaf/okl

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.