[Feature] Add RAG retrieval poisoning test scenarios
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- ai, security, testing-qa
Research direction
Start by reviewing related issue #2085 and the existing scenario library and attack executor. Add pyrit/scenario/scenarios/rag_retrieval_poisoning.py with predefined retrieval-poisoning attack vectors and scenario docstrings. The work is done when the scenarios are integrated with the existing executor and documented in the new file.
Written by the indexing model from the issue text.
Description
Summary
Add built-in test scenarios for RAG retrieval poisoning attacks to PyRIT's scenario library.
Problem
RAG systems are vulnerable to retrieval poisoning where malicious content is injected into the knowledge base. PyRIT currently lacks dedicated scenarios for testing this attack surface.
Proposed Implementation
- New file: pyrit/scenario/scenarios/rag_retrieval_poisoning.py
- Predefined attack vectors for retrieval poisoning
- Integration with existing attack executor
- Documentation in scenario docstrings
Use Case
Security teams can use these scenarios to test RAG-based LLM applications against known poisoning attack patterns.
References
- Related: issue #2085
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 896
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 155
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/PyRIT
-
Bug: triage help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Bug: triage
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
BUG: ScorerMetrics.to_json() raises TypeError on the trial_scores array ScorerEvaluator attaches Open
Difficulty 3/5 1-2 days Newbie friendliness 78/100
-
Bug: triage help wanted
Difficulty 3/5 1-2 days Newbie friendliness 75/100
-
Difficulty 4/5 3-5 days Newbie friendliness 68/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100