BUG JailBreakV-28K loader reads a `transfer_attack_type` column that does not exist upstream and drops the real `from` / `transfer_from_llm` provenance
维护者通常 2 天内回复
评估
- 难度
- 1/5
- 预计耗时
- 1-3 小时
- 新手友好度
- 35/100
- Issue 类型
- 缺陷
- 描述清晰度
- 描述清楚
- 活跃度
- 停滞
- 技术栈
- python
调研方向
Read pyrit/datasets/seed_datasets/remote/jailbreakv_28k_dataset.py and tests/unit/datasets/test_jailbreakv_28k_dataset.py, comparing the loader and fixture with the upstream columns listed in the issue. Confirm the metadata preserves from and boolean transfer_from_llm and omits transfer_attack_type; the issue says a fix with regression tests is ready and links open PR #3162.
由索引模型根据 Issue 内容生成。
描述
Describe the bug
_JailbreakV28KDataset (pyrit/datasets/seed_datasets/remote/jailbreakv_28k_dataset.py) copies per-row provenance into each seed's metadata:
row_metadata: dict[str, str | int] = {
"source_format": item.get("format", ""),
"transfer_attack_type": item.get("transfer_attack_type", ""),
"policy": raw_policy,
"image_path_raw": image_rel_path,
}
The upstream dataset has no transfer_attack_type column. Its columns, in both the JailBreakV_28K and mini_JailBreakV_28K splits, are:
id, jailbreak_query, redteam_query, format, policy, image_path, from, selected_mini, transfer_from_llm
(from https://datasets-server.huggingface.co/rows?dataset=JailbreakV-28K/JailBreakV-28k&config=JailBreakV_28K&split=mini_JailBreakV_28K; the HF commit history shows no data change since the 2024-06-22 upload). The dataset card documents from as the source of the row and transfer_from_llm as True when the jailbreak query was transferred from a text-only LLM attack, which is what distinguishes the ~20k LLM-transfer rows from the ~8k MLLM-specific rows.
Because .get(..., "") masks the missing key, every one of the 28,000 rows is emitted with metadata["transfer_attack_type"] == "", and the two columns that actually carry provenance are discarded. The sibling _JailbreakVRedTeam2KDataset loader keeps "from": item.get("from", "") correctly.
The unit test passes only because its fixture fabricates a transfer_attack_type column (tests/unit/datasets/test_jailbreakv_28k_dataset.py), so CI never sees the real column set. This is the same masking pattern as #2987.
Steps/Code to Reproduce
No network needed: feed the loader a row with the real upstream columns.
import pathlib
from unittest.mock import AsyncMock, patch
from pyrit.datasets.seed_datasets.remote.jailbreakv_28k_dataset import _JailbreakV28KDataset
tmp = pathlib.Path("/tmp/jbv")
(tmp / "JailBreakV_28k" / "llm_transfer_attack").mkdir(parents=True, exist_ok=True)
(tmp / "JailBreakV_28K.zip").write_bytes(b"fake")
(tmp / "JailBreakV_28k" / "llm_transfer_attack" / "SD_related_566.png").write_bytes(b"\x89PNG")
row = {
"id": 0, "jailbreak_query": "...", "redteam_query": "...", "format": "Template",
"policy": "Unethical Behavior", "image_path": "llm_transfer_attack/SD_related_566.png",
"from": "GPT Generate", "selected_mini": True, "transfer_from_llm": True,
}
loader = _JailbreakV28KDataset(zip_dir=str(tmp))
with patch.object(loader, "_fetch_from_huggingface_async", new=AsyncMock(return_value=[row])):
import asyncio
dataset = asyncio.run(loader.fetch_dataset_async())
print(dataset.seeds[0].metadata)
Expected Results
{'source_format': 'Template', 'from': 'GPT Generate', 'policy': 'Unethical Behavior', 'image_path_raw': 'llm_transfer_attack/SD_related_566.png', 'row_id': '0', 'transfer_from_llm': True}
Actual Results
{'source_format': 'Template', 'transfer_attack_type': '', 'policy': 'Unethical Behavior', 'image_path_raw': 'llm_transfer_attack/SD_related_566.png', 'row_id': '0'}
Suggested fix
Replace the phantom key with "from": item.get("from", ""), mirroring the RedTeam-2K loader, and keep transfer_from_llm as a boolean when the column is present. Update the test fixture to use the real column set and assert on from / transfer_from_llm, plus a regression assertion that transfer_attack_type is no longer emitted.
I have the fix with regression tests ready and can open the PR.
Versions
- OS: macOS 26.5
- Python version: 3.14.8
- PyRIT version: 1.2.0.dev0, installed from
main(93bf08e4) in editable mode
- 主要语言
- Python
- 星标
- 4.6k
- 派生
- 944
- 平均合并
- 2 天 20 小时
- 30 天内合并 PR
- 309
环境准备
在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 没有贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
microsoft/PyRIT 的其他 Issue
-
BUG UnicodeReplacementConverter writes characters outside the BMP as invalid five-digit \u escapes可能已有人在做 @ShivamSharma43 于 1 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 2 天内回复
-
BUG: label queries only return the final conversation of an attack可能已有人在做 @richlundeen 于 2 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 85/100
microsoft/PyRIT#3095 · 1 条评论 · 已指派 1 人 ·
维护者通常 2 天内回复
-
BUG Configuration keeps runtime-status errors after polling recovers可能已有人在做 @joeyvictorino 于 2 天前认领。 未关闭Bug: triage GUI help wanted
难度 2/5 1-3 小时 新手友好度 86/100
microsoft/PyRIT#2868 · 4 条评论 · 已指派 1 人 ·
维护者通常 2 天内回复
-
BUG QuestionAnswerScorer counts a wrong option as correct when the answer appears inside a longer word or number可能已有人在做 @barkha1research-ai 今天认领。 未关闭Bug: triage
难度 2/5 1-3 小时 新手友好度 25/100
维护者通常 2 天内回复
-
feature-request
难度 4/5 3-5 天 新手友好度 68/100
维护者通常 2 天内回复
相似的 Issue
-
bug difficulty:2 good first issue help wanted
难度 2/5 1-3 小时 新手友好度 88/100
MakazhanAlpamys/Soup#1865 ·
维护者通常 1 天内回复
-
area: agents area: docs good first issue
难度 2/5 1-3 小时 新手友好度 88/100
openapi/gknowledge#23 ·
-
useless-suppression reported for deprecated-* messages that should be exempt可能已有人在做 @adityasinha-ghub 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 88/100
pylint-dev/pylint#11656 ·
维护者通常 1 天内回复
-
area:provisioning effort:S priority:P2 provider:local-libvirt risk:daytime-only status:in-progress type:bug
难度 2/5 1-3 小时 新手友好度 78/100
randomparity/kdive#3203 · 2 条评论 ·
维护者通常 1 天内回复
-
get() for StrEnum可能已有人在做 @FeJo5952 今天认领。 未关闭good first issue
难度 2/5 1-3 小时 新手友好度 72/100
collective/icalendar#1921 · 1 条评论 ·
维护者通常 1 天内回复