Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

FEAT Proposal: red-team decision models as targets (decision flip-rate testing)

Đang mở
  • Có gì không ổn?
#3,246 2 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 2 ngày

@shashank03-dev đang làm issue này rồi.

Từ ngày 18/10/2026.

  • #3250 của @shashank03-dev — đang mở

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
15/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python
Lĩnh vực
ai, security, testing

Hướng nghiên cứu

This is a design proposal, not a scoped task. Read the open PR #3250 and the referenced #3188, which adds LiteLLMDecisionTarget and DecisionScorer, to see what already exists. Maintainers still need to answer whether decision-model red-teaming is in scope and whether state-only conversion should be a converter wrapper or attack-level request building. Work can start only after those answers and after #3188 merges.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Is your feature request related to a problem? Please describe.

Decision models (Microsoft-Decision-1, TypeSafe Jev, Perplexity Decider, Cloudflare Clef, Laya, and others) don't generate text. They take a state and typed questions and return a probability, a choice, or a score. They are being deployed as guardrails, routers, and graders, so a wrong decision is the failure that matters, not harmful text.

#3188 adds LiteLLMDecisionTarget and uses Microsoft-Decision-1 as a judge through DecisionScorer. Nothing in PyRIT treats a decision model as the system under test. Right now I can send a request to one, but I can't attack it in a useful way:

  • Converters run over the whole request, so anything other than a trivial converter breaks the {"state", "questions"} JSON that LiteLLMDecisionTarget expects. There is no way to perturb only the state.
  • No scorer reads a decision answer and reports whether it moved where the attacker wanted.
  • There is no shared metric to compare models or check vendor stability claims.

Published results suggest these models are easy to flip:

  • JevOut (arXiv 2609.30243, code released): natural-looking added context redirected 61.4-73.2% of initially correct decisions across four decision systems, including Jev, often with high confidence.
  • Type-Safe Is Not Error-Free (arXiv 2609.26758): reassigning option names to definitions raised flip rates by up to 70.4 points with yes/no names versus 0/1 names, at a 0% type-error rate.
  • Microsoft reports a 1.3% average flip rate for Microsoft-Decision-1 across eight perturbations. I haven't seen an independent check of that number.

This also affects PyRIT's own scoring. DecisionScorer puts the target's response into the decision model's state. If added context can flip a decision model, a target response can lower the score of an attack that actually succeeded.

Describe the solution you'd like

Add support for testing decision models as targets, building on LiteLLMDecisionTarget from #3188 so it works for every provider that LiteLLM's decision API supports. The metric is the decision flip rate: the share of initially correct decisions that change under a perturbation that shouldn't change the answer, measured above the model's rerun noise.

  1. State-only conversion. A converter wrapper (working name DecisionStateConverter) that parses the decision request, applies an existing converter to the state (or selected fields of a structured state), and re-serializes the request. This lets existing converters and single-turn attacks run against decision models unchanged.
  2. A decision answer scorer. A true/false scorer (working name DecisionFlipScorer) that reads the target's JSON answer and returns true when a noul crosses a threshold in the attacker's direction, or a choice/score answer lands on the attacker's chosen option.
  3. Probe families, each tied to a source:
    • instruction injection in the state
    • natural-context flipping (JevOut)
    • option-name polarity and reassignment (arXiv 2609.26758)
    • meaning-preserving perturbations: paraphrase, option reordering, formatting noise (matching Microsoft's published protocol)
    • existing PyRIT converters applied to the state
  4. Judge flipping. The same probes applied to DecisionScorer inputs, to measure how often a response that achieved the objective is scored below threshold once context is added.

I'd split this into separate PRs (1 and 2 first, then the probe families, then a scenario if wanted), each with unit tests and a doc page.

Before opening any PR, I'll run a pilot on Laya (open weights, run locally) and Microsoft-Decision-1, then on Jev and other models I can access. I'll post the flip rates per probe family here, with the rerun-noise baseline.

Describe alternatives you've considered, if relevant
  • A provider-specific target (JevTarget etc.). Not needed. LiteLLMDecisionTarget already covers Microsoft Foundry, TypeSafe, OpenAI, Perplexity, Cloudflare, Databricks, Fireworks, and self-hosted Laya through one model_name.
  • HTTPTarget with a JSON callback. Works for transport, but it doesn't solve state-only conversion or scoring decision answers.
  • SelectiveTextConverter. Its selection strategies work on raw text, so they can't address a field of a JSON request without risking invalid JSON.
  • Building the request in an attack after conversion, instead of a converter wrapper. This also works. I'd like a maintainer's view on which fits PyRIT better.
Additional context

How this relates to existing work:

  • #3188 uses a decision model as a judge. This proposal tests decision models as targets and reuses its target. It doesn't change LiteLLMDecisionTarget or DecisionScorer. I'd wait for #3188 to merge before starting.
  • #2168 measures whether rewording a scorer's rubric flips its verdict. The flip-rate idea is similar, but this proposal perturbs the input being judged, not the rubric, and targets decision models in general. I'll reuse its rerun-noise baseline approach where it fits so the two stay consistent.

Questions:

  1. Is red-teaming decision models in scope for PyRIT?
  2. Converter wrapper or attack-level request building for state-only conversion?
  3. Should the probe families ship as a scenario, or start as a notebook?

References:

Ngôn ngữ chính
Python
Star
4.6k
Fork
962
Merge trung bình
2 ngày 15 giờ
Pull request đã merge (30 ngày)
390

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

  • Không có Dockerfile hay tệp Docker Compose
  • Có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của microsoft/PyRIT

Tất cả issue của microsoft/PyRIT

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.