novelty: spread the atom cap across the artifact instead of its opening

Closed
  • Is something wrong?
#293 1 comment 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@antonwolfy is already working on this.

Since Nov 1, 2026.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
machine-learning, testing

Research direction

Start with the novelty analyze extraction and atom-cap selection flow, then read docs/skill-novelty-evidence.md and docs/reviewing.md to understand current tier measurements and bundle behavior. The issue leaves the sampling strata and tier evidence decisions open, so record those decisions before implementing. Done means capped tiers probe a deterministic spread sample, selection tests pin the behavior, and the docs and cap warning describe the rule.

Written by the indexing model from the issue text.

Description

enhancement

Problem

novelty analyze caps the atoms it probes (25 at --tier triage, 60 at standard). Extraction takes them in document order and stops at the cap, so a capped run measures only the opening of the artifact. Triage on IntelPython/dpnp#3099's guidance files:

file spans atoms from share of tokens
root AGENTS.md 61 s0002–s0012 about a quarter
dpnp/tests/AGENTS.md 32 s0003–s0013 about a third

The rules a reader most wanted judged (the leftover-reference rule, the changelog format, the CI table, ## Reviewing, the directory guides) were never measured.

A separate change makes the run say how far the cap reached. That doesn't solve the underlying problem: the capped rate describes whatever the file opens with. So two artifacts' rates mean different things when one opens with an introduction and the other with rules, and a file's rate moves when its sections are reordered.

Proposal

Extract every span, then probe max_atoms atoms spread over the file.

  • Selection: stratified by span position (or by tokens), with a fixed seed. sampling records the rule (cap-spread:<n>).
  • Cost: probing, the expensive part, stays at the cap. Extraction grows: 3 batches instead of 1 for the 61-span file. The whole triage run there was 103 calls and $0.78.
  • Bundles: with --include-resources, spread over the files as well, so references/ is reached on a skill whose SKILL.md alone has 25 atoms (today docs/reviewing.md says the flag reaches references/ only below the cap).

This changes what a tier measures, so it needs a research-register decision. The tier profiles in docs/skill-novelty-evidence.md were measured with document order.

Open questions

  • Strata: by spans, by tokens, or by headings?
  • Tier evidence: does it change the tier comparison in the evidence page enough to need re-measuring a subset?
  • Shortening: the shorten_artifact forfeit ("scored only against the capped atoms, i.e. the opening") changes. Do shortening candidates then cover the whole file?
  • Determinism and cache: the selection must be reproducible across runs and stable under the response cache.

Done when

A decision record exists, and triage (and the other capped tiers, if agreed) probe a spread sample. Tests pin the selection, the docs describe it, and the cap warning names the rule.

Dominant language
Python
Stars
9
Forks
9
Avg merge
1d 9h
Merged PRs (30d)
295

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from intel/dokimi

All issues in intel/dokimi

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.