BUG SelfAskScaleScorer crashes on an out-of-range judge score instead of retrying
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 82/100
Research direction
Start with SelfAskScaleScorer, SelfAskGeneralFloatScaleScorer, and the response-handler implementation next to TrueFalseResponseHandler. Trace where parsed scores enter Score construction, then verify that out-of-range judge values are handled within the retry path while valid values continue through normally.
Written by the indexing model from the issue text.
Description
On the default TAP 1–10 scale, if the judge answers 0 or 11, SelfAskScaleScorer fails after one call instead of retrying:
RuntimeError: Error in scorer SelfAskScaleScorer: ... Float scale scorers must have a score value between 0 and 1. Got -0.1111111111111111
The response handler only checks that the value is a finite float. The range check happens later, when the Score is built, which is outside the JSON retry. So one sloppy answer ends the attack, and this is the default scorer for TAP and Crescendo. SelfAskGeneralFloatScaleScorer has the same problem. Likert and the true/false scorers already validate their score domain inside the parser, so a bad value there just gets retried.
Suggested fix: a small NumericRangeResponseHandler wrapper next to TrueFalseResponseHandler, used by both scale scorers.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 896
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 191
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/PyRIT
-
BUG HarmBench loader drops ContextString, so contextual behaviors are sent without their context Open
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
Similar issues
-
agent-ready documentation needs-triage
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 91/100
-
workflow-status page template still says reusable workflows are "triggered only by workflow_call:" Open
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
instance instance add
Difficulty 1/5 Under an hour Newbie friendliness 72/100
searxng/searx-instances#939 · 1 comment ·
-
area-deployment area-integrations triage:bot-seen
Difficulty 2/5 Half a day Newbie friendliness 86/100