FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
Research direction
Start by inspecting PyRIT's existing dataset loaders and SeedDataset handling, then trace how metadata and filters are represented. Implement the requested train, validation, test, combined, attack, benign, and paired loading behavior with pair metadata preserved. Add mocked unit tests, documentation, and bibliography updates; verify scenario-isolated splits and typed filters.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem? Please describe.
PyRIT does not currently expose a paired prompt-injection dataset designed to test the boundary between legitimate agent workflows and adversarial instructions. Attack-only collections are useful for recall, but they do not let users build or inspect false-positive-aware test surfaces where benign and attack prompts intentionally share the same scenario and vocabulary.
Describe the solution you'd like
Add a remote loader for the CC BY 4.0 Agentic Prompt-Injection Boundary Pairs dataset.
The proposed loader would:
- load the published train, validation, test, or combined surface;
- default to attack examples for red-team use while allowing benign or paired loading;
- preserve
pair_id,scenario_id,attack_family,pair_family,source_context,risk_domain,target_boundary,split, and label metadata; - support typed filters for label, split, attack family, and source context;
- keep scenario-isolated split semantics intact;
- include mocked unit tests, documentation, and bibliography updates.
The dataset contains 1,200 English records organized as 600 controlled benign/attack pairs across 50 enterprise and agentic scenarios and 12 attack families. JSONL and Parquet artifacts, deterministic build scripts, validation, checksums, and a 30% stratified manual-review record are public.
Describe alternatives you've considered, if relevant
Users can download the JSONL files directly, but that bypasses PyRIT dataset discovery, metadata preservation, filtering, and consistent SeedDataset handling. A generic file loader would also lose the paired-boundary semantics unless every consumer rebuilt the same mapping.
Additional context
Dataset: https://huggingface.co/datasets/3nesdeniz/agentic-prompt-injection-boundary-pairs
Source and build pipeline: https://github.com/3nesdeniz/agentic-prompt-injection-boundary-pairs
I maintain the dataset and can keep the loader schema synchronized with tagged dataset releases.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 896
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 155
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/PyRIT
-
Bug: triage help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Bug: triage
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
BUG: ScorerMetrics.to_json() raises TypeError on the trial_scores array ScorerEvaluator attaches Open
Difficulty 3/5 1-2 days Newbie friendliness 78/100
-
Bug: triage help wanted
Difficulty 3/5 1-2 days Newbie friendliness 75/100
-
Difficulty 4/5 3-5 days Newbie friendliness 68/100
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100