Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

FEAT Proposal: red-team decision models as targets (decision flip-rate testing)

Abierto
  • ¿Algo no está bien?
#3,246 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

@shashank03-dev ya está trabajando en esto.

Desde el 15/10/2026.

  • #3250 de @shashank03-dev — abierto

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
15/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
ai, security, testing

Línea de trabajo

This is a design proposal, not a scoped task. Read the open PR #3250 and the referenced #3188, which adds LiteLLMDecisionTarget and DecisionScorer, to see what already exists. Maintainers still need to answer whether decision-model red-teaming is in scope and whether state-only conversion should be a converter wrapper or attack-level request building. Work can start only after those answers and after #3188 merges.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Is your feature request related to a problem? Please describe.

Decision models (Microsoft-Decision-1, TypeSafe Jev, Perplexity Decider, Cloudflare Clef, Laya, and others) don't generate text. They take a state and typed questions and return a probability, a choice, or a score. They are being deployed as guardrails, routers, and graders, so a wrong decision is the failure that matters, not harmful text.

#3188 adds LiteLLMDecisionTarget and uses Microsoft-Decision-1 as a judge through DecisionScorer. Nothing in PyRIT treats a decision model as the system under test. Right now I can send a request to one, but I can't attack it in a useful way:

  • Converters run over the whole request, so anything other than a trivial converter breaks the {"state", "questions"} JSON that LiteLLMDecisionTarget expects. There is no way to perturb only the state.
  • No scorer reads a decision answer and reports whether it moved where the attacker wanted.
  • There is no shared metric to compare models or check vendor stability claims.

Published results suggest these models are easy to flip:

  • JevOut (arXiv 2609.30243, code released): natural-looking added context redirected 61.4-73.2% of initially correct decisions across four decision systems, including Jev, often with high confidence.
  • Type-Safe Is Not Error-Free (arXiv 2609.26758): reassigning option names to definitions raised flip rates by up to 70.4 points with yes/no names versus 0/1 names, at a 0% type-error rate.
  • Microsoft reports a 1.3% average flip rate for Microsoft-Decision-1 across eight perturbations. I haven't seen an independent check of that number.

This also affects PyRIT's own scoring. DecisionScorer puts the target's response into the decision model's state. If added context can flip a decision model, a target response can lower the score of an attack that actually succeeded.

Describe the solution you'd like

Add support for testing decision models as targets, building on LiteLLMDecisionTarget from #3188 so it works for every provider that LiteLLM's decision API supports. The metric is the decision flip rate: the share of initially correct decisions that change under a perturbation that shouldn't change the answer, measured above the model's rerun noise.

  1. State-only conversion. A converter wrapper (working name DecisionStateConverter) that parses the decision request, applies an existing converter to the state (or selected fields of a structured state), and re-serializes the request. This lets existing converters and single-turn attacks run against decision models unchanged.
  2. A decision answer scorer. A true/false scorer (working name DecisionFlipScorer) that reads the target's JSON answer and returns true when a noul crosses a threshold in the attacker's direction, or a choice/score answer lands on the attacker's chosen option.
  3. Probe families, each tied to a source:
    • instruction injection in the state
    • natural-context flipping (JevOut)
    • option-name polarity and reassignment (arXiv 2609.26758)
    • meaning-preserving perturbations: paraphrase, option reordering, formatting noise (matching Microsoft's published protocol)
    • existing PyRIT converters applied to the state
  4. Judge flipping. The same probes applied to DecisionScorer inputs, to measure how often a response that achieved the objective is scored below threshold once context is added.

I'd split this into separate PRs (1 and 2 first, then the probe families, then a scenario if wanted), each with unit tests and a doc page.

Before opening any PR, I'll run a pilot on Laya (open weights, run locally) and Microsoft-Decision-1, then on Jev and other models I can access. I'll post the flip rates per probe family here, with the rerun-noise baseline.

Describe alternatives you've considered, if relevant
  • A provider-specific target (JevTarget etc.). Not needed. LiteLLMDecisionTarget already covers Microsoft Foundry, TypeSafe, OpenAI, Perplexity, Cloudflare, Databricks, Fireworks, and self-hosted Laya through one model_name.
  • HTTPTarget with a JSON callback. Works for transport, but it doesn't solve state-only conversion or scoring decision answers.
  • SelectiveTextConverter. Its selection strategies work on raw text, so they can't address a field of a JSON request without risking invalid JSON.
  • Building the request in an attack after conversion, instead of a converter wrapper. This also works. I'd like a maintainer's view on which fits PyRIT better.
Additional context

How this relates to existing work:

  • #3188 uses a decision model as a judge. This proposal tests decision models as targets and reuses its target. It doesn't change LiteLLMDecisionTarget or DecisionScorer. I'd wait for #3188 to merge before starting.
  • #2168 measures whether rewording a scorer's rubric flips its verdict. The flip-rate idea is similar, but this proposal perturbs the input being judged, not the rubric, and targets decision models in general. I'll reuse its rerun-noise baseline approach where it fits so the two stay consistent.

Questions:

  1. Is red-teaming decision models in scope for PyRIT?
  2. Converter wrapper or attack-level request building for state-only conversion?
  3. Should the probe families ship as a scenario, or start as a notebook?

References:

Lenguaje dominante
Python
Estrellas
4.6k
Forks
944
Merge medio
2 d 18 h
PR fusionados (30 d)
358

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/PyRIT

Todos los issues de microsoft/PyRIT

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.