Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Suggestion: Built-in test scenarios for RAG retrieval poisoning

Open
#2,085 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Quiet
Tech stack
python
Domain
ai, security

Research direction

Start by reviewing PyRIT's existing test-scenario modules and the vector-store or RAG integration points relevant to retrieval. Define how poisoned document injection and its effect on generated responses should be simulated and measured; done means the framework includes a documented, repeatable scenario for evaluating retrieval poisoning.

Written by the indexing model from the issue text.

Description

PyRIT currently focuses primarily on prompt-level attacks against LLMs, but doesn't include pre-built scenarios for testing RAG-specific vulnerabilities, such as injecting adversarial content into a vector store to manipulate retrieval results (retrieval poisoning). Adding a dedicated test module for simulating poisoned document injection and measuring its effect on generated responses would extend PyRIT's coverage to a growing attack surface as RAG architectures become standard in enterprise AI deployments.

Dominant language
Python
Stars
4.5k
Forks
896
Avg merge
3d 8h
Merged PRs (30d)
191

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/PyRIT

All issues in microsoft/PyRIT

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.