Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

BUG JailBreakV-28K loader reads a `transfer_attack_type` column that does not exist upstream and drops the real `from` / `transfer_from_llm` provenance

Abierto
  • ¿Algo no está bien?
#3,161 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

@kushaltalati ya está trabajando en esto.

Desde el 12/10/2026.

  • #3162 de @kushaltalati — abierto

Evaluación

Dificultad
1/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
python
Área
ai-infra-agents

Línea de trabajo

Read pyrit/datasets/seed_datasets/remote/jailbreakv_28k_dataset.py and tests/unit/datasets/test_jailbreakv_28k_dataset.py, comparing the loader and fixture with the upstream columns listed in the issue. Confirm the metadata preserves from and boolean transfer_from_llm and omits transfer_attack_type; the issue says a fix with regression tests is ready and links open PR #3162.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

_JailbreakV28KDataset (pyrit/datasets/seed_datasets/remote/jailbreakv_28k_dataset.py) copies per-row provenance into each seed's metadata:

row_metadata: dict[str, str | int] = {
    "source_format": item.get("format", ""),
    "transfer_attack_type": item.get("transfer_attack_type", ""),
    "policy": raw_policy,
    "image_path_raw": image_rel_path,
}

The upstream dataset has no transfer_attack_type column. Its columns, in both the JailBreakV_28K and mini_JailBreakV_28K splits, are:

id, jailbreak_query, redteam_query, format, policy, image_path, from, selected_mini, transfer_from_llm

(from https://datasets-server.huggingface.co/rows?dataset=JailbreakV-28K/JailBreakV-28k&config=JailBreakV_28K&split=mini_JailBreakV_28K; the HF commit history shows no data change since the 2024-06-22 upload). The dataset card documents from as the source of the row and transfer_from_llm as True when the jailbreak query was transferred from a text-only LLM attack, which is what distinguishes the ~20k LLM-transfer rows from the ~8k MLLM-specific rows.

Because .get(..., "") masks the missing key, every one of the 28,000 rows is emitted with metadata["transfer_attack_type"] == "", and the two columns that actually carry provenance are discarded. The sibling _JailbreakVRedTeam2KDataset loader keeps "from": item.get("from", "") correctly.

The unit test passes only because its fixture fabricates a transfer_attack_type column (tests/unit/datasets/test_jailbreakv_28k_dataset.py), so CI never sees the real column set. This is the same masking pattern as #2987.

Steps/Code to Reproduce

No network needed: feed the loader a row with the real upstream columns.

import pathlib
from unittest.mock import AsyncMock, patch

from pyrit.datasets.seed_datasets.remote.jailbreakv_28k_dataset import _JailbreakV28KDataset

tmp = pathlib.Path("/tmp/jbv")
(tmp / "JailBreakV_28k" / "llm_transfer_attack").mkdir(parents=True, exist_ok=True)
(tmp / "JailBreakV_28K.zip").write_bytes(b"fake")
(tmp / "JailBreakV_28k" / "llm_transfer_attack" / "SD_related_566.png").write_bytes(b"\x89PNG")

row = {
    "id": 0, "jailbreak_query": "...", "redteam_query": "...", "format": "Template",
    "policy": "Unethical Behavior", "image_path": "llm_transfer_attack/SD_related_566.png",
    "from": "GPT Generate", "selected_mini": True, "transfer_from_llm": True,
}
loader = _JailbreakV28KDataset(zip_dir=str(tmp))
with patch.object(loader, "_fetch_from_huggingface_async", new=AsyncMock(return_value=[row])):
    import asyncio
    dataset = asyncio.run(loader.fetch_dataset_async())
print(dataset.seeds[0].metadata)
Expected Results
{'source_format': 'Template', 'from': 'GPT Generate', 'policy': 'Unethical Behavior', 'image_path_raw': 'llm_transfer_attack/SD_related_566.png', 'row_id': '0', 'transfer_from_llm': True}
Actual Results
{'source_format': 'Template', 'transfer_attack_type': '', 'policy': 'Unethical Behavior', 'image_path_raw': 'llm_transfer_attack/SD_related_566.png', 'row_id': '0'}
Suggested fix

Replace the phantom key with "from": item.get("from", ""), mirroring the RedTeam-2K loader, and keep transfer_from_llm as a boolean when the column is present. Update the test fixture to use the real column set and assert on from / transfer_from_llm, plus a regression assertion that transfer_attack_type is no longer emitted.

I have the fix with regression tests ready and can open the PR.

Versions
  • OS: macOS 26.5
  • Python version: 3.14.8
  • PyRIT version: 1.2.0.dev0, installed from main (93bf08e4) in editable mode
Lenguaje dominante
Python
Estrellas
4.6k
Forks
944
Merge medio
2 d 20 h
PR fusionados (30 d)
309

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/PyRIT

Todos los issues de microsoft/PyRIT

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.