Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

FEAT Add support for the HarmEval dataset

Cerrado
#2,770 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
55/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
huggingface, python
Área
data, documentation, machine-learning, testing

Línea de trabajo

Start by inspecting PyRIT's existing dataset-provider patterns and the published Safe_Infer_Final_Dataset_550_Without_Answer.csv with authorized Hugging Face access. Confirm the prompt and category fields, then follow the existing registration, mocked-test, attribution, documentation, and citation conventions. Done means the gated loader supports authentication without bundling data and clearly documents access and usage terms.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

datasets help wanted
Is your feature request related to a problem? Please describe.

SoftMINER-Group/HarmEval is relevant to PyRIT but does not currently have a built-in dataset loader. The authors describe approximately 550 harmful queries across 11 policy categories. Hugging Face access is gated, which should be supported rather than excluding the dataset.

This is a dedicated follow-up to #429.

Describe the solution you'd like

Add a remote seed dataset loader for the official HarmEval release, following the existing dataset-provider patterns.

  • Inspect the actual data with authorized access and confirm the prompt and category fields. The column names suggested in #429 have not been verified. The published file is Safe_Infer_Final_Dataset_550_Without_Answer.csv.
  • Support authenticated access and document the access requirements, including accepting the upstream conditions and providing a Hugging Face token. Do not bundle the gated data in this repository.
  • Preserve source attribution and any available category metadata, using the appropriate PyRIT seed types.
  • Add mocked loader tests, register the dataset, and update the paired dataset documentation and paper citation.

The scope is dataset integration, not implementing SafeInfer's decoding method.

Describe alternatives you've considered, if relevant

Users with access can import the CSV manually. A built-in loader would make the dataset easier to discover and use consistently with PyRIT's other datasets.

Additional context

Please confirm and document the applicable usage terms: the Hugging Face metadata declares Apache-2.0, while the access form also includes a non-commercial-use-only condition. This needs clarification rather than assuming unrestricted use from the license tag alone.

Lenguaje dominante
Python
Estrellas
4.5k
Forks
896
Merge medio
3 d 8 h
PR fusionados (30 d)
191

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/PyRIT

Todos los issues de microsoft/PyRIT

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.