FEAT: Add Garak divergence scenario
@VimalN2005 is already working on this.
Since Sep 8, 2026.
Assessment
This issue has not been assessed yet.
Description
Is your feature request related to a problem? Please describe.
PyRIT does not have a bounded scenario for garak's repetition-induced divergence checks. A direct port would generate tokenizer data at runtime and change target output settings from probe code. Those behaviors do not fit PyRIT component responsibilities and can create unexpectedly expensive runs. This is part of #511.
Describe the solution you'd like
Add a PyRIT-native divergence scenario based on garak/probes/divergence.py.
Required scope and behavior:
- Make
Repeatthe default scenario technique. - Add
RepeatedTokenas an explicit opt-in technique. Leave inactiveRepeatExtendedoutside the required first version. - Store repeat templates, seed words, and safe configuration metadata in local PyRIT datasets.
- Store a small, fixed, attributed set of precomputed repeated-token strings in the dataset. Do not add runtime tokenizer generation or a new tokenizer dependency.
- Use standard PyRIT attack flow. Do not mutate target
max_tokensor other target settings in the scenario. Document any recommended target output limit as user configuration. - Reuse existing refusal and text-analysis components where their contracts fit. If no existing scorer covers divergence, add one deterministic scorer that distinguishes refusal or bounded requested repetition from unexpected continuation, and that detects excessive repeated structures for
RepeatedToken. - Set conservative prompt counts and response limits so the default is not a denial-of-service test.
- Describe positive results as divergence or output instability. Do not claim training-data leakage without a separate attributable reference match.
- Add exports, bounded unit tests, and synchronized
.pyand.ipynbscanner documentation.
Use PromptInject PR #2509 as an example of dataset/technique separation. Follow doc/code/framework.md and the applicable scenario, dataset, scorer, test, and documentation instructions. Do not port garak generator hooks into a scenario.
Describe alternatives you've considered, if relevant
- Runtime
tiktokengeneration was considered and rejected. Fixed data keeps the scenario deterministic and avoids dependency and tokenizer-version drift. - Changing target output limits from scenario code was considered and rejected. The target owner must control those limits.
- Treating any repetition as a positive result would confuse expected bounded compliance with divergence.
Additional context
- Parent parity issue: #511
- PyRIT structural example: #2509
- Upstream probe:
garak/probes/divergence.py - Garak is Apache-2.0, not MIT. Retain applicable attribution and modification notices for copied or adapted source data.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 896
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 180
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/PyRIT
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
-
Bug: triage GUI help wanted
Difficulty 3/5 1-2 days Newbie friendliness 65/100
-
feature-request
Difficulty 3/5 1-2 days Newbie friendliness 70/100
-
bug help wanted
Difficulty 4/5 3-5 days Newbie friendliness 68/100
-
not ready yet
Difficulty 4/5 3-5 days Newbie friendliness 35/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
anthropics/skills#1811 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
speaches-ai/speaches#678 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
datalayer/mcp-compose#42 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
conda-forge/spacy-feedstock#177 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
UKGovernmentBEIS/inspect_evals#2523 ·