Buff._derive_new_attempt shares notes and detector_results dicts between attempts, so sibling attempts report the last attempt's scores
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 86/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- ai, testing-qa
Research direction
Read garak/buffs/base.py, especially Buff._derive_new_attempt, and the related tests in tests/buffs/test_buffs.py. Run the buff tests and reproduce the sibling-attempt case described in the issue. Done means derived attempts retain independent notes and detector_results, with a regression test covering the original and sibling scores.
Written by the indexing model from the issue text.
Description
Steps to reproduce
garak.buffs.base.Buff._derive_new_attempt builds the derived attempt with notes=source_attempt.notes and detector_results=source_attempt.detector_results, and Attempt.__init__ stores those objects as-is (self.notes = {} if notes is None else notes). Every attempt derived from one source therefore shares a single notes dict and a single detector_results dict with the source and with each other. No API key or model is needed to see it; this uses only buffs.lowercase.Lowercase and probes.test.Blank:
from garak import _config, _plugins
import garak.attempt
_config.load_base_config()
_config.plugins.buffs_include_original_prompt = True
_config.buffmanager.buffs = [_plugins.load_plugin("buffs.lowercase.Lowercase")]
probe = _plugins.load_plugin("probes.test.Blank")
original = garak.attempt.Attempt(prompt=garak.attempt.Message("HELLO", lang="en"), probe_classname="test.Blank")
attempts = probe._buff_hook([original])
print("attempts:", len(attempts), [a.prompt.last_message().text for a in attempts])
orig, buffed = attempts
print("notes shared:", orig.notes is buffed.notes)
print("detector_results shared:", orig.detector_results is buffed.detector_results)
print("original attempt notes:", orig.notes)
# what Harness._run_detector does, once per attempt
orig.detector_results["always.Fail"] = [1.0]
buffed.detector_results["always.Fail"] = [0.0]
print("original attempt's recorded score:", orig.detector_results["always.Fail"])
Output on main (2212c73e, 0.16.1.pre1):
attempts: 2 ['HELLO', 'hello']
notes shared: True
detector_results shared: True
original attempt notes: {'buff_creator': 'Lowercase', 'buff_source_attempt_uuid': '18a05f20-...', 'buff_source_seq': -1}
original attempt's recorded score: [0.0]
Expected behavior
Each attempt owns its notes and detector_results. The original attempt should carry no buff_creator / buff_source_* metadata, and a score written for one attempt should not change what another attempt reports.
Current behavior
Harness._run_detectorwritesattempt.detector_results[detector_name] = list(...)once per attempt, andEvaluator._evaluate_one_detectorreadsattempt.detector_results[detector_name]afterwards. With the shared dict, every attempt in a sibling group ends up reporting the last sibling's scores. This affects any run withbuffs_include_original_prompt: true(original vs. buffed attempt collide) and every multi-yield buff regardless of that setting:buffs.paraphrase.PegasusT5/Fastyield the original plus N paraphrases from one_derive_new_attemptchain,buffs.low_resource_languages.LRLBuffyields one attempt per language. The hitlog and pass/fail counts for those attempts are therefore wrong, in whichever direction the last sibling happened to land.- The original attempt's
notesare polluted withbuff_creator,buff_source_attempt_uuid,buff_source_seq(visible inreport.jsonl), and per-attempt notes written insidetransform(e.g.LRL_buff_dest_lang) are overwritten by later siblings.
garak version
main at 2212c73e (garak --version -> 0.16.1.pre1); also present at v0.16.0.
Additional Information
- macOS 15 (arm64), CPU only
- Python 3.12.13
- direct repository checkout with
git,pip install -e .[tests] - Repro above; no target run required
- The fix is to copy
notesanddetector_resultsin_derive_new_attempt(copy.deepcopy), which I will open a PR for together with a regression test intests/buffs/test_buffs.py.
- Dominant language
- Python
- Stars
- 9.3k
- Forks
- 1.3k
- Avg merge
- 4d 18h
- Merged PRs (30d)
- 12
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/garak
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100