FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader

Open
#2,174 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
ai, data, security

Research direction

Start by inspecting PyRIT's existing dataset loaders and SeedDataset handling, then trace how metadata and filters are represented. Implement the requested train, validation, test, combined, attack, benign, and paired loading behavior with pair metadata preserved. Add mocked unit tests, documentation, and bibliography updates; verify scenario-isolated splits and typed filters.

Written by the indexing model from the issue text.

Description

Is your feature request related to a problem? Please describe.

PyRIT does not currently expose a paired prompt-injection dataset designed to test the boundary between legitimate agent workflows and adversarial instructions. Attack-only collections are useful for recall, but they do not let users build or inspect false-positive-aware test surfaces where benign and attack prompts intentionally share the same scenario and vocabulary.

Describe the solution you'd like

Add a remote loader for the CC BY 4.0 Agentic Prompt-Injection Boundary Pairs dataset.

The proposed loader would:

  • load the published train, validation, test, or combined surface;
  • default to attack examples for red-team use while allowing benign or paired loading;
  • preserve pair_id, scenario_id, attack_family, pair_family, source_context, risk_domain, target_boundary, split, and label metadata;
  • support typed filters for label, split, attack family, and source context;
  • keep scenario-isolated split semantics intact;
  • include mocked unit tests, documentation, and bibliography updates.

The dataset contains 1,200 English records organized as 600 controlled benign/attack pairs across 50 enterprise and agentic scenarios and 12 attack families. JSONL and Parquet artifacts, deterministic build scripts, validation, checksums, and a 30% stratified manual-review record are public.

Describe alternatives you've considered, if relevant

Users can download the JSONL files directly, but that bypasses PyRIT dataset discovery, metadata preservation, filtering, and consistent SeedDataset handling. A generic file loader would also lose the paired-boundary semantics unless every consumer rebuilt the same mapping.

Additional context

Dataset: https://huggingface.co/datasets/3nesdeniz/agentic-prompt-injection-boundary-pairs

Source and build pipeline: https://github.com/3nesdeniz/agentic-prompt-injection-boundary-pairs

I maintain the dataset and can keep the loader schema synchronized with tagged dataset releases.

Dominant language
Python
Stars
4.5k
Forks
896
Avg merge
3d 7h
Merged PRs (30d)
155

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/PyRIT

All issues in microsoft/PyRIT

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.