Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

FEAT Add support for the HarmEval dataset

Closed
#2,770 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
huggingface, python

Research direction

Start by inspecting PyRIT's existing dataset-provider patterns and the published Safe_Infer_Final_Dataset_550_Without_Answer.csv with authorized Hugging Face access. Confirm the prompt and category fields, then follow the existing registration, mocked-test, attribution, documentation, and citation conventions. Done means the gated loader supports authentication without bundling data and clearly documents access and usage terms.

Written by the indexing model from the issue text.

Description

datasets help wanted
Is your feature request related to a problem? Please describe.

SoftMINER-Group/HarmEval is relevant to PyRIT but does not currently have a built-in dataset loader. The authors describe approximately 550 harmful queries across 11 policy categories. Hugging Face access is gated, which should be supported rather than excluding the dataset.

This is a dedicated follow-up to #429.

Describe the solution you'd like

Add a remote seed dataset loader for the official HarmEval release, following the existing dataset-provider patterns.

  • Inspect the actual data with authorized access and confirm the prompt and category fields. The column names suggested in #429 have not been verified. The published file is Safe_Infer_Final_Dataset_550_Without_Answer.csv.
  • Support authenticated access and document the access requirements, including accepting the upstream conditions and providing a Hugging Face token. Do not bundle the gated data in this repository.
  • Preserve source attribution and any available category metadata, using the appropriate PyRIT seed types.
  • Add mocked loader tests, register the dataset, and update the paired dataset documentation and paper citation.

The scope is dataset integration, not implementing SafeInfer's decoding method.

Describe alternatives you've considered, if relevant

Users with access can import the CSV manually. A built-in loader would make the dataset easier to discover and use consistently with PyRIT's other datasets.

Additional context

Please confirm and document the applicable usage terms: the Hugging Face metadata declares Apache-2.0, while the access form also includes a non-commercial-use-only condition. This needs clarification rather than assuming unrestricted use from the license tag alone.

Dominant language
Python
Stars
4.5k
Forks
896
Avg merge
3d 8h
Merged PRs (30d)
191

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/PyRIT

All issues in microsoft/PyRIT

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.