BUG HarmBench loader drops ContextString, so contextual behaviors are sent without their context
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 74/100
Research direction
Locate the HarmBench loader and read baselines/direct_request/direct_request.py, which shows the reference input combination. Verify that contextual behaviors reach the target with both ContextString and Behavior, then run the relevant HarmBench or loader tests; done means contextual objectives include their context without breaking non-contextual behaviors.
Written by the indexing model from the issue text.
Description
100 of the 400 HarmBench behaviors are "contextual" and only make sense with their ContextString ("Modify this blind SQL injection code...", "Given this section of the Wikipedia article..."). The loader only uses Behavior, so the context ends up in metadata that nothing reads, and a quarter of HarmBench objectives point at text the target never sees. HarmBench is the default dataset for a few scenarios (jailbreak, multilingual, red team agent, the adversarial benchmark), so this comes up a lot.
HarmBench itself combines them as f"{context_str}\n\n---\n\n{behavior}" in baselines/direct_request/direct_request.py. I'd suggest doing the same.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 896
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 191
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/PyRIT
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
stephrobert/dsoxlab#238 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
sublimehq/package_control#1780 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
nwg-piotr/nwg-displays#145 ·