Parent: LLM Vulnerability Scanner Feature Parity with Garak, Giskard, CyberSecEval

Open
#511 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Quiet
Tech stack
python
Domain
security

Research direction

No implementation file, test, or entry point is identified. Start by comparing attack scenarios supported by Garak, Giskard, and CyberSecEval with PyRIT's documented capabilities, then open a focused follow-up issue for one missing scenario; done means the scenario has a clear implementation or documentation scope.

Written by the indexing model from the issue text.

Description

good first issue help wanted

TLDR; We want help creating issues where Garak, Giskard, or CyberSecEval support an attack scenario where PyRIT doesn't support it.

Background

Other tools like Garak, Giskard, and CyberSecEval have different use cases from PyRIT, but there is overlap as well. If there is an attack in one of these other platforms, we probably want to support it also and would love issues being created so we can track this (or clarify if it already exists/improve our docs).

This paper tackles a really difficult problem, and that's comparing these tools (and PyRIT). When I was looking, some of the things I noticed that it mentioned PyRIT was lacking were:

  • Lack of multi-language support (I think we actually have this as a converter)
  • Lack of GCG (we have this as an auxiliary module)
  • Dan, AutoDan, etc (I think we have support for all of this. We certainly have DAN and ways to modify those prompts with GPTFuzz - but we would love to know what we're missing)
  • Insecure Coding Test (we don't have this, but it would be an easy scorer to write!)

And there might be more that I missed!

So as an example of issues we'd like, all of the above are good to open. Even if we think we support them (like in the multi-language case) we can dive into specifics to make sure we're not missing anything, and potentially improve documentation so it's more obvious these scenarios are supported.

Describe the solution you'd like

Please open follow-up sub-issues with any features you notice or find in these other tools that PyRIT doesn't have! That way we can track them. Also, we would love community to tackle any of these also, but it still can be helpful to open an issue so we can help guide :D

Dominant language
Python
Stars
4.5k
Forks
896
Avg merge
3d 9h
Merged PRs (30d)
170

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/PyRIT

All issues in microsoft/PyRIT

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.