FEAT Proposal: red-team decision models as targets (decision flip-rate testing)

This is a design proposal, not a scoped task. Read the open PR #3250 and the referenced #3188, which adds LiteLLMDecisionTarget and DecisionScorer, to see what already exists. Maintainers still need to answer whether decision-model red-teaming is in scope and whether state-only conversion should be a converter wrapper or attack-level request building. Work can start only after those answers and after #3188 merges.

Is your feature request related to a problem? Please describe.

Decision models (Microsoft-Decision-1, TypeSafe Jev, Perplexity Decider, Cloudflare Clef, Laya, and others) don't generate text. They take a state and typed questions and return a probability, a choice, or a score. They are being deployed as guardrails, routers, and graders, so a wrong decision is the failure that matters, not harmful text.

#3188 adds LiteLLMDecisionTarget and uses Microsoft-Decision-1 as a judge through DecisionScorer. Nothing in PyRIT treats a decision model as the system under test. Right now I can send a request to one, but I can't attack it in a useful way:

• Converters run over the whole request, so anything other than a trivial converter breaks the {"state", "questions"} JSON that LiteLLMDecisionTarget expects. There is no way to perturb only the state.
• No scorer reads a decision answer and reports whether it moved where the attacker wanted.
• There is no shared metric to compare models or check vendor stability claims.

Published results suggest these models are easy to flip:

• JevOut (arXiv 2609.30243, code released): natural-looking added context redirected 61.4-73.2% of initially correct decisions across four decision systems, including Jev, often with high confidence.
• Type-Safe Is Not Error-Free (arXiv 2609.26758): reassigning option names to definitions raised flip rates by up to 70.4 points with yes/no names versus 0/1 names, at a 0% type-error rate.
• Microsoft reports a 1.3% average flip rate for Microsoft-Decision-1 across eight perturbations. I haven't seen an independent check of that number.

This also affects PyRIT's own scoring. DecisionScorer puts the target's response into the decision model's state. If added context can flip a decision model, a target response can lower the score of an attack that actually succeeded.

Describe the solution you'd like

Add support for testing decision models as targets, building on LiteLLMDecisionTarget from #3188 so it works for every provider that LiteLLM's decision API supports. The metric is the decision flip rate: the share of initially correct decisions that change under a perturbation that shouldn't change the answer, measured above the model's rerun noise.

1. State-only conversion. A converter wrapper (working name DecisionStateConverter) that parses the decision request, applies an existing converter to the state (or selected fields of a structured state), and re-serializes the request. This lets existing converters and single-turn attacks run against decision models unchanged.
2. A decision answer scorer. A true/false scorer (working name DecisionFlipScorer) that reads the target's JSON answer and returns true when a noul crosses a threshold in the attacker's direction, or a choice/score answer lands on the attacker's chosen option.
3. Probe families, each tied to a source:
   • instruction injection in the state
   • natural-context flipping (JevOut)
   • option-name polarity and reassignment (arXiv 2609.26758)
   • meaning-preserving perturbations: paraphrase, option reordering, formatting noise (matching Microsoft's published protocol)
   • existing PyRIT converters applied to the state
4. Judge flipping. The same probes applied to DecisionScorer inputs, to measure how often a response that achieved the objective is scored below threshold once context is added.

I'd split this into separate PRs (1 and 2 first, then the probe families, then a scenario if wanted), each with unit tests and a doc page.

Before opening any PR, I'll run a pilot on Laya (open weights, run locally) and Microsoft-Decision-1, then on Jev and other models I can access. I'll post the flip rates per probe family here, with the rerun-noise baseline.

Describe alternatives you've considered, if relevant

• A provider-specific target (JevTarget etc.). Not needed. LiteLLMDecisionTarget already covers Microsoft Foundry, TypeSafe, OpenAI, Perplexity, Cloudflare, Databricks, Fireworks, and self-hosted Laya through one model_name.
• HTTPTarget with a JSON callback. Works for transport, but it doesn't solve state-only conversion or scoring decision answers.
• SelectiveTextConverter. Its selection strategies work on raw text, so they can't address a field of a JSON request without risking invalid JSON.
• Building the request in an attack after conversion, instead of a converter wrapper. This also works. I'd like a maintainer's view on which fits PyRIT better.

Additional context

How this relates to existing work:

• #3188 uses a decision model as a judge. This proposal tests decision models as targets and reuses its target. It doesn't change LiteLLMDecisionTarget or DecisionScorer. I'd wait for #3188 to merge before starting.
• #2168 measures whether rewording a scorer's rubric flips its verdict. The flip-rate idea is similar, but this proposal perturbs the input being judged, not the rubric, and targets decision models in general. I'll reuse its rerun-noise baseline approach where it fits so the two stay consistent.

Questions:

1. Is red-teaming decision models in scope for PyRIT?
2. Converter wrapper or attack-level request building for state-only conversion?
3. Should the probe families ship as a scenario, or start as a notebook?

References:

• JevOut: Natural Context Can Flip Decision Models, https://arxiv.org/abs/2609.30243
• Type-Safe Is Not Error-Free, https://arxiv.org/abs/2609.26758
• Microsoft-Decision-1 announcement, https://commandline.microsoft.com/microsoft-decision-1-model-foundry/
• LiteLLM decision API, https://docs.litellm.ai/docs/decisions