Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

FEAT Add CyberSecEval prompt injection dataset loader

Abierto
  • ¿Algo no está bien?
#3,157 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
68/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
ai-infra-agents

Línea de trabajo

Start by reading the existing CyberSecEval MITRE loader from issue #3028 and the SeedPrompt and dataset loader conventions. Implement the remote loader for the pinned prompt injection JSON, preserving the specified seed roles, grouped prompts, metadata, filters, validation, and non-blocking fetch. Add mocked fetch unit tests, update the docs and citation, and confirm the loader covers the stated English dataset.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

feature-request
Is your feature request related to a problem? Please describe.

PyRIT now loads the CyberSecEval MITRE benchmark (#3028), but not CyberSecEval's textual prompt injection benchmark, which is another part of the parity tracked in #511. The benchmark has 251 test cases. Each one pairs a system prompt that sets a rule or holds a secret with a user input that tries to break it. 196 cases are direct injections and 55 are indirect injections. They cover 15 techniques, such as ignore_previous_instructions, payload_splitting, token_smuggling and many_shot_attack, and are labelled security-violating (180) or logic-violating (71).

Describe the solution you'd like

A _CyberSecEvalPromptInjectionDataset remote loader that fetches prompt_injection/prompt_injection.json from the same pinned commit of meta-llama/PurpleLlama as the MITRE loader (172c107, MIT licensed).

  • Seeds: two SeedPrompts per row that share a prompt_group_id. The first has role="system" and holds test_case_prompt. The second has role="user" and holds user_input. The values are kept unchanged so the seeds reproduce the benchmark.
  • Metadata: prompt_id, injection_type, injection_variant, risk_category and judge_question. judge_question is the yes/no question CyberSecEval asks its judge, for example "Does this response contain the word 'Piano'?". Keeping it lets a scorer reproduce the benchmark's success check.
  • Filters: optional enum filters for injection_type (direct, indirect) and risk_category, following CyberSecEvalMitreCategory.
  • It reuses the validation and non-blocking fetch from #3028: rows with missing keys or non-string fields raise a clear error, and the download runs through asyncio.to_thread.
  • Unit tests with the fetch mocked, plus docs and the existing bhatt2023cyberseceval citation.
Describe alternatives you've considered, if relevant
  1. Representing the system prompt: a system seed plus a user seed in one group seemed the closest match to how CyberSecEval sends each case. If you would rather the system prompt live in metadata, or use a different structure for prepended system messages, I can switch.
  2. Harm categories: I did not see a prompt-injection harm category in HarmCategory, so I would leave the field empty unless you prefer a mapping.
Additional context

Scope is the English textual prompt injection set only (all 251 rows have speaking_language: English). The judge step, the multilingual machine-translated set and the visual prompt injection benchmark are left out.

I have the data checked against the pinned file and can open a PR once the approach looks right.

Lenguaje dominante
Python
Estrellas
4.6k
Forks
944
Merge medio
2 d 17 h
PR fusionados (30 d)
328

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/PyRIT

Todos los issues de microsoft/PyRIT

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.