BUG: attacks record FAILURE when the objective scorer returns no score
Maintainers usually reply within 2 days
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 45/100
Research direction
Start with _determine_attack_outcome in prompt_sending.py, then check the matching logic in multi_prompt_sending.py, red_teaming.py around line 411, and chunked_request.py, where a missing score maps to FAILURE. Crescendo and TAP already raise in this case, so read them for the expected handling. Done means that when a scorer is configured, a response exists, and the score is None, the outcome is UNDETERMINED with the message 'Objective scorer returned no score', with tests covering the modality-mismatch case. The maintainer filed this to settle the outcome semantics first, so confirm the approach in the thread before changing code.
Written by the indexing model from the issue text.
Description
When the objective scorer returns no score, several attacks record FAILURE, which reads as "measured and defended". AttackOutcome.UNDETERMINED exists for exactly this case.
prompt_sending.py_determine_attack_outcome:if response: return FAILUREmulti_prompt_sending.py: samered_teaming.py:411:... if context.last_score else AttackOutcome.FAILUREchunked_request.py: "No score returned from scorer" maps to FAILURE
This happens e.g. when a text-only scorer gets an image response. Crescendo and TAP raise in the same situation instead, so the attacks don't agree either.
PromptSendingAttack: outcome=FAILURE score=None reason='Failed to achieve objective after 1 attempts'
MultiPromptSendingAttack: outcome=FAILURE score=None reason='Failed to achieve objective'
It understates ASR for modality-mismatched runs, since unscored rows count as defended. Proposal: when a scorer is configured, a response exists and the score is None, return UNDETERMINED with "Objective scorer returned no score". Filing first since it changes reported outcomes.
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 944
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 309
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/PyRIT
-
BUG UnicodeReplacementConverter writes characters outside the BMP as invalid five-digit \u escapesPossibly taken @ShivamSharma43 claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
microsoft/PyRIT#3095 · 3 comments ·
Maintainers usually reply within 2 days
-
BUG Configuration keeps runtime-status errors after polling recoversPossibly taken @joeyvictorino claimed this 3 days ago. OpenBug: triage GUI help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
microsoft/PyRIT#2868 · 4 comments · 1 assignee ·
Maintainers usually reply within 2 days
-
BUG: MemoryInterface.get_request_from_response fails with IndexError for non-zero starting sequence or sequence gapsPossibly taken @RohithPariki claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
Maintainers usually reply within 2 days
-
Conversation editor never revokes blob URLs of removed attachmentsPossibly taken @ManoharPaturi claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
Maintainers usually reply within 2 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
task
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
danparshall/claude_researcher#87 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
datajoint/datajoint-python#1586 ·
Maintainers usually reply within 1 day
-
1D intercept of length 1 is accepted as `fix_params` in population GLM with more than 1 unitPossibly taken @dhvani7014 claimed this today. Openbug good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
flatironinstitute/nemos#642 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
uniswap-python/uniswap-python#487 ·