BUG: attacks record FAILURE when the objective scorer returns no score
维护者通常 2 天内回复
评估
调研方向
Start with _determine_attack_outcome in prompt_sending.py, then check the matching logic in multi_prompt_sending.py, red_teaming.py around line 411, and chunked_request.py, where a missing score maps to FAILURE. Crescendo and TAP already raise in this case, so read them for the expected handling. Done means that when a scorer is configured, a response exists, and the score is None, the outcome is UNDETERMINED with the message 'Objective scorer returned no score', with tests covering the modality-mismatch case. The maintainer filed this to settle the outcome semantics first, so confirm the approach in the thread before changing code.
由索引模型根据 Issue 内容生成。
描述
When the objective scorer returns no score, several attacks record FAILURE, which reads as "measured and defended". AttackOutcome.UNDETERMINED exists for exactly this case.
prompt_sending.py_determine_attack_outcome:if response: return FAILUREmulti_prompt_sending.py: samered_teaming.py:411:... if context.last_score else AttackOutcome.FAILUREchunked_request.py: "No score returned from scorer" maps to FAILURE
This happens e.g. when a text-only scorer gets an image response. Crescendo and TAP raise in the same situation instead, so the attacks don't agree either.
PromptSendingAttack: outcome=FAILURE score=None reason='Failed to achieve objective after 1 attempts'
MultiPromptSendingAttack: outcome=FAILURE score=None reason='Failed to achieve objective'
It understates ASR for modality-mismatched runs, since unscored rows count as defended. Proposal: when a scorer is configured, a response exists and the score is None, return UNDETERMINED with "Objective scorer returned no score". Filing first since it changes reported outcomes.
- 主要语言
- Python
- 星标
- 4.6k
- 派生
- 944
- 平均合并
- 2 天 20 小时
- 30 天内合并 PR
- 309
环境准备
在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 没有贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
microsoft/PyRIT 的其他 Issue
-
BUG UnicodeReplacementConverter writes characters outside the BMP as invalid five-digit \u escapes可能已有人在做 @ShivamSharma43 于 2 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 2 天内回复
-
BUG: label queries only return the final conversation of an attack可能已有人在做 @richlundeen 于 2 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 85/100
microsoft/PyRIT#3095 · 1 条评论 · 已指派 1 人 ·
维护者通常 2 天内回复
-
BUG Configuration keeps runtime-status errors after polling recovers可能已有人在做 @joeyvictorino 于 2 天前认领。 未关闭Bug: triage GUI help wanted
难度 2/5 1-3 小时 新手友好度 86/100
microsoft/PyRIT#2868 · 4 条评论 · 已指派 1 人 ·
维护者通常 2 天内回复
-
BUG JailBreakV-28K loader reads a `transfer_attack_type` column that does not exist upstream and drops the real `from` / `transfer_from_llm` provenance可能已有人在做 @kushaltalati 今天认领。 未关闭
难度 1/5 1-3 小时 新手友好度 35/100
microsoft/PyRIT#3161 · 1 条评论 ·
维护者通常 2 天内回复
-
BUG QuestionAnswerScorer counts a wrong option as correct when the answer appears inside a longer word or number可能已有人在做 @barkha1research-ai 今天认领。 未关闭Bug: triage
难度 2/5 1-3 小时 新手友好度 25/100
维护者通常 2 天内回复
相似的 Issue
-
effort: low prio: low ready
难度 2/5 1-3 小时 新手友好度 82/100
digitalfabrik/integreat-cms#4616 ·
维护者通常 5 天内回复
-
bug difficulty:2 good first issue help wanted
难度 2/5 1-3 小时 新手友好度 88/100
MakazhanAlpamys/Soup#1865 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
FinanceFlash/unvibecode#233 ·
维护者通常 1 天内回复
-
area: agents area: docs good first issue
难度 2/5 1-3 小时 新手友好度 88/100
openapi/gknowledge#23 ·
-
useless-suppression reported for deprecated-* messages that should be exempt可能已有人在做 @adityasinha-ghub 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 88/100
pylint-dev/pylint#11656 ·
维护者通常 1 天内回复