Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

FEAT Add support for the HarmEval dataset

Chiusa
#2,770 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
huggingface, python
Ambito
data, documentation, machine-learning, testing

Direzione di ricerca

Start by inspecting PyRIT's existing dataset-provider patterns and the published Safe_Infer_Final_Dataset_550_Without_Answer.csv with authorized Hugging Face access. Confirm the prompt and category fields, then follow the existing registration, mocked-test, attribution, documentation, and citation conventions. Done means the gated loader supports authentication without bundling data and clearly documents access and usage terms.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

datasets help wanted
Is your feature request related to a problem? Please describe.

SoftMINER-Group/HarmEval is relevant to PyRIT but does not currently have a built-in dataset loader. The authors describe approximately 550 harmful queries across 11 policy categories. Hugging Face access is gated, which should be supported rather than excluding the dataset.

This is a dedicated follow-up to #429.

Describe the solution you'd like

Add a remote seed dataset loader for the official HarmEval release, following the existing dataset-provider patterns.

  • Inspect the actual data with authorized access and confirm the prompt and category fields. The column names suggested in #429 have not been verified. The published file is Safe_Infer_Final_Dataset_550_Without_Answer.csv.
  • Support authenticated access and document the access requirements, including accepting the upstream conditions and providing a Hugging Face token. Do not bundle the gated data in this repository.
  • Preserve source attribution and any available category metadata, using the appropriate PyRIT seed types.
  • Add mocked loader tests, register the dataset, and update the paired dataset documentation and paper citation.

The scope is dataset integration, not implementing SafeInfer's decoding method.

Describe alternatives you've considered, if relevant

Users with access can import the CSV manually. A built-in loader would make the dataset easier to discover and use consistently with PyRIT's other datasets.

Additional context

Please confirm and document the applicable usage terms: the Hugging Face metadata declares Apache-2.0, while the access form also includes a non-commercial-use-only condition. This needs clarification rather than assuming unrestricted use from the license tag alone.

Lingua principale
Python
Stelle
4.5k
Fork
896
Merge medio
3g 8h
PR unite (30g)
191

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/PyRIT

Tutte le issue di microsoft/PyRIT

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.