Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

FEAT Proposal: red-team decision models as targets (decision flip-rate testing)

Aperta
  • Qualcosa non va?
#3,246 2 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@shashank03-dev ci sta già lavorando.

Dal 18/10/2026.

  • #3250 di @shashank03-dev — aperta

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
15/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai, security, testing

Direzione di ricerca

This is a design proposal, not a scoped task. Read the open PR #3250 and the referenced #3188, which adds LiteLLMDecisionTarget and DecisionScorer, to see what already exists. Maintainers still need to answer whether decision-model red-teaming is in scope and whether state-only conversion should be a converter wrapper or attack-level request building. Work can start only after those answers and after #3188 merges.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Is your feature request related to a problem? Please describe.

Decision models (Microsoft-Decision-1, TypeSafe Jev, Perplexity Decider, Cloudflare Clef, Laya, and others) don't generate text. They take a state and typed questions and return a probability, a choice, or a score. They are being deployed as guardrails, routers, and graders, so a wrong decision is the failure that matters, not harmful text.

#3188 adds LiteLLMDecisionTarget and uses Microsoft-Decision-1 as a judge through DecisionScorer. Nothing in PyRIT treats a decision model as the system under test. Right now I can send a request to one, but I can't attack it in a useful way:

  • Converters run over the whole request, so anything other than a trivial converter breaks the {"state", "questions"} JSON that LiteLLMDecisionTarget expects. There is no way to perturb only the state.
  • No scorer reads a decision answer and reports whether it moved where the attacker wanted.
  • There is no shared metric to compare models or check vendor stability claims.

Published results suggest these models are easy to flip:

  • JevOut (arXiv 2609.30243, code released): natural-looking added context redirected 61.4-73.2% of initially correct decisions across four decision systems, including Jev, often with high confidence.
  • Type-Safe Is Not Error-Free (arXiv 2609.26758): reassigning option names to definitions raised flip rates by up to 70.4 points with yes/no names versus 0/1 names, at a 0% type-error rate.
  • Microsoft reports a 1.3% average flip rate for Microsoft-Decision-1 across eight perturbations. I haven't seen an independent check of that number.

This also affects PyRIT's own scoring. DecisionScorer puts the target's response into the decision model's state. If added context can flip a decision model, a target response can lower the score of an attack that actually succeeded.

Describe the solution you'd like

Add support for testing decision models as targets, building on LiteLLMDecisionTarget from #3188 so it works for every provider that LiteLLM's decision API supports. The metric is the decision flip rate: the share of initially correct decisions that change under a perturbation that shouldn't change the answer, measured above the model's rerun noise.

  1. State-only conversion. A converter wrapper (working name DecisionStateConverter) that parses the decision request, applies an existing converter to the state (or selected fields of a structured state), and re-serializes the request. This lets existing converters and single-turn attacks run against decision models unchanged.
  2. A decision answer scorer. A true/false scorer (working name DecisionFlipScorer) that reads the target's JSON answer and returns true when a noul crosses a threshold in the attacker's direction, or a choice/score answer lands on the attacker's chosen option.
  3. Probe families, each tied to a source:
    • instruction injection in the state
    • natural-context flipping (JevOut)
    • option-name polarity and reassignment (arXiv 2609.26758)
    • meaning-preserving perturbations: paraphrase, option reordering, formatting noise (matching Microsoft's published protocol)
    • existing PyRIT converters applied to the state
  4. Judge flipping. The same probes applied to DecisionScorer inputs, to measure how often a response that achieved the objective is scored below threshold once context is added.

I'd split this into separate PRs (1 and 2 first, then the probe families, then a scenario if wanted), each with unit tests and a doc page.

Before opening any PR, I'll run a pilot on Laya (open weights, run locally) and Microsoft-Decision-1, then on Jev and other models I can access. I'll post the flip rates per probe family here, with the rerun-noise baseline.

Describe alternatives you've considered, if relevant
  • A provider-specific target (JevTarget etc.). Not needed. LiteLLMDecisionTarget already covers Microsoft Foundry, TypeSafe, OpenAI, Perplexity, Cloudflare, Databricks, Fireworks, and self-hosted Laya through one model_name.
  • HTTPTarget with a JSON callback. Works for transport, but it doesn't solve state-only conversion or scoring decision answers.
  • SelectiveTextConverter. Its selection strategies work on raw text, so they can't address a field of a JSON request without risking invalid JSON.
  • Building the request in an attack after conversion, instead of a converter wrapper. This also works. I'd like a maintainer's view on which fits PyRIT better.
Additional context

How this relates to existing work:

  • #3188 uses a decision model as a judge. This proposal tests decision models as targets and reuses its target. It doesn't change LiteLLMDecisionTarget or DecisionScorer. I'd wait for #3188 to merge before starting.
  • #2168 measures whether rewording a scorer's rubric flips its verdict. The flip-rate idea is similar, but this proposal perturbs the input being judged, not the rubric, and targets decision models in general. I'll reuse its rerun-noise baseline approach where it fits so the two stay consistent.

Questions:

  1. Is red-teaming decision models in scope for PyRIT?
  2. Converter wrapper or attack-level request building for state-only conversion?
  3. Should the probe families ship as a scenario, or start as a notebook?

References:

Lingua principale
Python
Stelle
4.6k
Fork
962
Merge medio
2g 15h
PR unite (30g)
390

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/PyRIT

Tutte le issue di microsoft/PyRIT

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.