Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

BUG JailBreakV-28K loader reads a `transfer_attack_type` column that does not exist upstream and drops the real `from` / `transfer_from_llm` provenance

Geschlossen
  • Stimmt etwas nicht?
#3,161 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 2 Tagen

Ein zugehöriger Pull Request wurde bereits gemerged.

  • #3162 von @kushaltalati — gemerged

Bewertung

Schwierigkeit
1/5
Geschätzter Aufwand
1-3 Stunden
Anfängerfreundlichkeit
35/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Veraltet
Tech-Stack
python
Bereich
ai-infra-agents

Rechercherichtung

Read pyrit/datasets/seed_datasets/remote/jailbreakv_28k_dataset.py and tests/unit/datasets/test_jailbreakv_28k_dataset.py, comparing the loader and fixture with the upstream columns listed in the issue. Confirm the metadata preserves from and boolean transfer_from_llm and omits transfer_attack_type; the issue says a fix with regression tests is ready and links open PR #3162.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Describe the bug

_JailbreakV28KDataset (pyrit/datasets/seed_datasets/remote/jailbreakv_28k_dataset.py) copies per-row provenance into each seed's metadata:

row_metadata: dict[str, str | int] = {
    "source_format": item.get("format", ""),
    "transfer_attack_type": item.get("transfer_attack_type", ""),
    "policy": raw_policy,
    "image_path_raw": image_rel_path,
}

The upstream dataset has no transfer_attack_type column. Its columns, in both the JailBreakV_28K and mini_JailBreakV_28K splits, are:

id, jailbreak_query, redteam_query, format, policy, image_path, from, selected_mini, transfer_from_llm

(from https://datasets-server.huggingface.co/rows?dataset=JailbreakV-28K/JailBreakV-28k&config=JailBreakV_28K&split=mini_JailBreakV_28K; the HF commit history shows no data change since the 2024-06-22 upload). The dataset card documents from as the source of the row and transfer_from_llm as True when the jailbreak query was transferred from a text-only LLM attack, which is what distinguishes the ~20k LLM-transfer rows from the ~8k MLLM-specific rows.

Because .get(..., "") masks the missing key, every one of the 28,000 rows is emitted with metadata["transfer_attack_type"] == "", and the two columns that actually carry provenance are discarded. The sibling _JailbreakVRedTeam2KDataset loader keeps "from": item.get("from", "") correctly.

The unit test passes only because its fixture fabricates a transfer_attack_type column (tests/unit/datasets/test_jailbreakv_28k_dataset.py), so CI never sees the real column set. This is the same masking pattern as #2987.

Steps/Code to Reproduce

No network needed: feed the loader a row with the real upstream columns.

import pathlib
from unittest.mock import AsyncMock, patch

from pyrit.datasets.seed_datasets.remote.jailbreakv_28k_dataset import _JailbreakV28KDataset

tmp = pathlib.Path("/tmp/jbv")
(tmp / "JailBreakV_28k" / "llm_transfer_attack").mkdir(parents=True, exist_ok=True)
(tmp / "JailBreakV_28K.zip").write_bytes(b"fake")
(tmp / "JailBreakV_28k" / "llm_transfer_attack" / "SD_related_566.png").write_bytes(b"\x89PNG")

row = {
    "id": 0, "jailbreak_query": "...", "redteam_query": "...", "format": "Template",
    "policy": "Unethical Behavior", "image_path": "llm_transfer_attack/SD_related_566.png",
    "from": "GPT Generate", "selected_mini": True, "transfer_from_llm": True,
}
loader = _JailbreakV28KDataset(zip_dir=str(tmp))
with patch.object(loader, "_fetch_from_huggingface_async", new=AsyncMock(return_value=[row])):
    import asyncio
    dataset = asyncio.run(loader.fetch_dataset_async())
print(dataset.seeds[0].metadata)
Expected Results
{'source_format': 'Template', 'from': 'GPT Generate', 'policy': 'Unethical Behavior', 'image_path_raw': 'llm_transfer_attack/SD_related_566.png', 'row_id': '0', 'transfer_from_llm': True}
Actual Results
{'source_format': 'Template', 'transfer_attack_type': '', 'policy': 'Unethical Behavior', 'image_path_raw': 'llm_transfer_attack/SD_related_566.png', 'row_id': '0'}
Suggested fix

Replace the phantom key with "from": item.get("from", ""), mirroring the RedTeam-2K loader, and keep transfer_from_llm as a boolean when the column is present. Update the test fixture to use the real column set and assert on from / transfer_from_llm, plus a regression assertion that transfer_attack_type is no longer emitted.

I have the fix with regression tests ready and can open the PR.

Versions
  • OS: macOS 26.5
  • Python version: 3.14.8
  • PyRIT version: 1.2.0.dev0, installed from main (93bf08e4) in editable mode
Vorherrschende Sprache
Python
Sterne
4.6k
Forks
944
Ø Merge
2 T. 16 Std.
Gemergte PRs (30 T.)
350

Entwicklungsumgebung

In Codespaces öffnen

Startet den Dev-Container des Projekts im Browser, mit Ihrem eigenen GitHub-Konto.

  • Kein Dockerfile und keine Docker-Compose-Datei
  • Hat eine Pull-Request-Vorlage
  • Kein Beitragsleitfaden

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus microsoft/PyRIT

Alle Issues in microsoft/PyRIT

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.