Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

novelty: bad LLM replies cached or misreported (judge retry, conflict silent drop, retry warning) — follow-up to #40

Open
  • Is something wrong?
#219 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
ai, testing-qa

Research direction

Start with dokimi/novelty/classify.py for judge_value and detect_conflicts, then read dokimi/novelty/atoms.py for extraction retries and warnings. Update the related wording in novelty.v1.json and skill-novelty.md; use the stub transport probe described in the issue to check malformed rows, retries, and cache acceptance. Done means unusable replies are retried and excluded from the cache, dropped conflict rows are reported, and the max-token hint appears only for parse failures.

Written by the indexing model from the issue text.

Description

Follow-up to #40 (merged). Three places in the novelty pipeline where a bad LLM reply is either cached as a real result or reported with the wrong cause. All three were reproduced with a stub transport on main (5a133b2) and on the #40 head (3848e20); the probe script is at the end.

1. judge_value: a reply where every row is malformed is not retried, and it gets cached

Setup: 10 atoms, batch=5, and a judge that answers each row with needed: "false", or leaves out needed, or leaves out n.

5a133b2 (before #40) 3848e20 (#40 head)
needed: "false" 10 verdicts, all needed=True (bool("false")) 0 verdicts, 10 counted as unjudged_skipped
needed missing 10 verdicts, needed=False 0 verdicts, 10 unjudged_skipped
calls / retried batches 2 / 0 2 / 0
would the response cache store it (accept(reply)) yes yes

#40 stopped the wrong verdicts, but a judge reply with no usable row is neither split nor re-asked. The cache stores it, so every later run replays it. The warning says the atoms were "skipped by the judge's reply", but the reply could not be read at all. Extraction already handles the same case through _NoUsableRows (split the batch, keep the reply out of the cache). The judge parser has no equivalent check.

Suggested fix: if a judge reply has no row with a valid n and a real boolean needed, raise the same kind of "no usable rows" error, so the batch is split and the reply is not cached. Report malformed rows apart from rows the judge actually skipped, and fix the docstring at classify.py:118-119 and the warning at :154.

2. detect_conflicts: unreadable rows are dropped without being counted (already on main)

Setup: the reply is {"conflicts": [...]} and holds only rows that cannot be used: a == b, rows with no a/b, non-dict rows, or an index out of range.

  • On both commits: conflicts=0, pairs_compared=45/45, unchecked=0, no warning, and the cache would store the reply. The report says every pair was checked and none conflict.
  • The only difference is the self-pair: 5a133b2 reported a false conflict for it, and 3848e20 drops it silently.

Suggested fix: count the rows that were dropped and warn about them. If every row in a non-empty reply was dropped, treat the reply as unusable (no cache, retry) instead of as "no conflicts".

3. Extraction: the retry warning gives --max-tokens advice for replies that parsed fine (new in #40)

Setup: every row parses as JSON but none is usable ([{"foo": 1}, ...]).

  • 5a133b2: 0 atoms, 2 calls, no split. The reply was silently taken as empty, which is what #40 fixes.
  • 3848e20: the first warning is right ("replied with rows but none usable"). The second still says "re-asked 8 unparseable batches … a higher --max-tokens avoids the extra calls". These batches did parse, so a higher --max-tokens would not change anything.

Suggested fix: count retries by cause (parse failure vs. no usable rows) and give the --max-tokens hint only for parse failures. Update the same wording in the retried_batches docstring (atoms.py:189), the comment at :363-366, novelty.v1.json:295/306/313 and skill-novelty.md:799-800/817.

Repro

Run with PYTHONPATH=<checkout>/dokimi python probe.py, once on each commit. The script calls no LLM: it replaces LLMClient._default_transport and records what accept() returns, which is what the response cache uses to decide whether to store a reply.

probe.py 
"""Stub-transport probes for the pr-review-toolkit findings on dokimi PR 40.
Run with PYTHONPATH=<checkout>/dokimi on main (5a133b2) and the PR head (3848e20)."""
import json, re
from dokimi.novelty.atoms import ExtractionReport, extract_atoms
from dokimi.novelty.client import LLMClient
from dokimi.novelty.spans import segment
from dokimi.novelty import classify
from dokimi.novelty.classify import ClassifyReport, judge_value, detect_conflicts

DOC = "\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 11))
def ids(p): return re.findall(r"^\[(s\d+)\]", p, flags=re.M)
def good(s): return {"statement": f"{s} x", "question": f"q {s}?", "span_ids": [s]}

def client(reply_fn, role):
    """Stub client; also records what the response cache would be told (cache_if = accept)."""
    c = LLMClient(model="x", role=role)
    c.seen = []
    def fake(prompt, system, *, accept=None):
        r = reply_fn(prompt)
        c.seen.append(None if accept is None else bool(accept(r)))
        return r, None
    c._default_transport = fake
    return c

atoms = extract_atoms(segment(DOC), client=client(lambda p: json.dumps([good(s) for s in ids(p)]), "extractor"), batch_spans=5)
print("atoms", len(atoms))

print("\n== F1 judge: every row malformed ==")
for name, row in {"needed:'false'": lambda n: {"n": n, "needed": "false"},
                  "needed missing": lambda n: {"n": n, "verdict": "no"},
                  "n missing": lambda n: {"needed": True}}.items():
    c = client(lambda p: json.dumps([row(i + 1) for i in range(len(re.findall(r"^\d+\. ", p, flags=re.M)))]), "judge")
    rep = ClassifyReport()
    try:
        v = judge_value(atoms, client=c, task="t", batch=5, report=rep)
        print(f"  {name:15} verdicts={len(v)} needed={[x.needed for x in v.values()][:3]} unjudged={len(rep.unjudged)} "
              f"skipped={rep.unjudged_skipped} calls={rep.calls} retried={rep.retried_batches} cached={c.seen}")
        for w in rep.warnings(len(atoms)) if hasattr(rep, "warnings") else []: print("     W:", w[:220])
    except Exception as e:
        print(f"  {name:15} RAISE {type(e).__name__}: {e}")

print("\n== F2 detect_conflicts: only rows that cannot be read ==")
for name, rows in {"self-pair a==b": [{"a": 1, "b": 1, "why": "w"}],
                   "no a/b": [{"x": 1}], "non-dict": ["junk", 3],
                   "out of range": [{"a": 1, "b": 99}]}.items():
    c = client(lambda p, rows=rows: json.dumps({"conflicts": rows}), "conflict")
    rep = ClassifyReport()
    try:
        out = detect_conflicts(atoms, client=c, batch=10, report=rep)
        print(f"  {name:15} conflicts={len(out)} pairs={rep.pairs_compared}/{rep.pairs_total} unchecked={len(rep.unchecked)} "
              f"calls={rep.calls} cached={c.seen}")
        for w in rep.warnings(len(atoms)) if hasattr(rep, "warnings") else []: print("     W:", w[:220])
    except Exception as e:
        print(f"  {name:15} RAISE {type(e).__name__}: {e}")

print("\n== F3 extraction warning when rows parse but none is usable ==")
for name, f in {"all rows junk": lambda p: [{"foo": 1} for _ in ids(p)],
                "truncated JSON (real parse failure)": None}.items():
    if f is None:
        fn = lambda p: '[{"statement": "cut'
    else:
        fn = lambda p, f=f: json.dumps(f(p))
    rep = ExtractionReport()
    try:
        a = extract_atoms(segment(DOC), client=client(fn, "extractor"), batch_spans=5, report=rep)
        print(f"  {name}: atoms={len(a)} calls={rep.calls} retried={rep.retried_batches}")
    except Exception as e:
        print(f"  {name}: RAISE {type(e).__name__} calls={rep.calls} retried={rep.retried_batches}")
    for w in rep.warnings(10): print("     W:", w[:400])

These were found by an automated review (pr-review-toolkit) of #40 and then confirmed by hand with the stubs above.

Dominant language
Python
Stars
8
Forks
7
Avg merge
1d 4h
Merged PRs (30d)
199

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from intel/dokimi

All issues in intel/dokimi

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.