BUG: Tone, Tense, Translation, Noise, ScientificTranslation and Variation converter identifiers ignore a custom prompt_template
维护者通常 2 天内回复
评估
调研方向
The issue identifies six converters in PyRIT: ToneConverter, TenseConverter, TranslationConverter, NoiseConverter, ScientificTranslationConverter, and VariationConverter. Read their identifier-building methods and compare them with LLMGenericTextConverter and IPAConverter, then inspect identifier tests. Done means custom prompt templates produce distinct identifiers from the corresponding built-in templates; linked pull request #3316 shows this work is already under way.
由索引模型根据 Issue 内容生成。
描述
Describe the bug
Six LLM-based converters accept a custom prompt_template but leave it out of their identifiers, so a converter with a custom template has the same hash as one with the built-in template:
ToneConverter(onlytone)TenseConverter(onlytense)TranslationConverter(onlylanguage)NoiseConverter(onlynoiseandnumber_errors)ScientificTranslationConverter(onlymode, so a built-in mode with a custom template looks like the built-in one)VariationConverter(no params at all)
The template is the system prompt sent to the converter target, so it changes the output. LLMGenericTextConverter already identifies its templates with system_prompt_template_hash and user_prompt_template_hash, and IPAConverter keeps them by building on super()._build_identifier(), but these six subclasses replace the base params instead.
Because the default registry name comes from that hash, registering a converter with a custom template next to the default one raises a duplicate-name error, and memory and analytics cannot tell the two apart. This is the same problem as #3287, for the LLM-based converters.
Steps/Code to Reproduce
import asyncio
from pyrit.converter import (
NoiseConverter,
ScientificTranslationConverter,
TenseConverter,
ToneConverter,
TranslationConverter,
VariationConverter,
)
from pyrit.models import SeedPrompt
from pyrit.prompt_target import OpenAIChatTarget
from pyrit.setup import IN_MEMORY, initialize_pyrit_async
asyncio.run(initialize_pyrit_async(memory_db_type=IN_MEMORY))
target = OpenAIChatTarget(endpoint="https://example.invalid/openai/v1", api_key="unused", model_name="gpt-4o")
def custom(text: str, *params: str) -> SeedPrompt:
return SeedPrompt(value=text, data_type="text", parameters=list(params))
pairs = [
(
ToneConverter(converter_target=target, tone="angry"),
ToneConverter(converter_target=target, tone="angry", prompt_template=custom("Rewrite in a {{ tone }} tone, in French.", "tone")),
),
(
TenseConverter(converter_target=target, tense="past"),
TenseConverter(converter_target=target, tense="past", prompt_template=custom("Use the {{ tense }} tense, as a poem.", "tense")),
),
(
TranslationConverter(converter_target=target, language="French"),
TranslationConverter(converter_target=target, language="French", prompt_template=custom("Translate to {{ language }} slang.", "language")),
),
(
NoiseConverter(converter_target=target),
NoiseConverter(converter_target=target, prompt_template=custom("Add {{ number_errors }} typos: {{ noise }}", "noise", "number_errors")),
),
(
ScientificTranslationConverter(converter_target=target, mode="academic"),
ScientificTranslationConverter(converter_target=target, mode="academic", prompt_template=custom("Rewrite as a lab report.")),
),
(
VariationConverter(converter_target=target),
VariationConverter(converter_target=target, prompt_template=custom("Give {{ number_iterations }} rhyming variations.", "number_iterations")),
),
]
for default, custom_one in pairs:
print(default.get_identifier().unique_name, custom_one.get_identifier().unique_name)
Expected Results
Each pair has two different identifiers.
Actual Results
ToneConverter::71c29a7e ToneConverter::71c29a7e
TenseConverter::b9a79a58 TenseConverter::b9a79a58
TranslationConverter::04918fb0 TranslationConverter::04918fb0
NoiseConverter::a5ec8804 NoiseConverter::a5ec8804
ScientificTranslationConverter::a303aaf0 ScientificTranslationConverter::a303aaf0
VariationConverter::46dc63ec VariationConverter::46dc63ec
Screenshots
N/A
Versions
- OS: Windows 11
- Python version: 3.11.9
- PyRIT version: installed from main (f277fedd) in editable mode
- 主要语言
- Python
- 星标
- 4.6k
- 派生
- 962
- 平均合并
- 2 天 16 小时
- 30 天内合并 PR
- 354
环境准备
在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 没有贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
microsoft/PyRIT 的其他 Issue
-
BUG: label queries only return the final conversation of an attack可能已有人在做 @u7k4rs6 于 5 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 85/100
microsoft/PyRIT#3095 · 4 条评论 · 已指派 1 人 ·
维护者通常 2 天内回复
-
难度 3/5 1-2 天 新手友好度 72/100
microsoft/PyRIT#3325 · 1 条评论 ·
维护者通常 2 天内回复
-
Detection coverage gaps: 47 missed payload classes across 9 output scorers可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭
难度 4/5 3-5 天 新手友好度 53/100
维护者通常 2 天内回复
-
难度 4/5 3-5 天 新手友好度 46/100
维护者通常 2 天内回复
-
BUG: DenylistConverter and TransparencyAttackConverter identifiers ignore settings that change output可能已有人在做 @ShivamSharma43 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 35/100
维护者通常 2 天内回复
相似的 Issue
-
debt
难度 2/5 1-3 小时 新手友好度 68/100
microsoft/vscode-flake8#601 ·
维护者通常 1 天内回复
-
nonews refactor
难度 2/5 1-3 小时 新手友好度 75/100
Uninett/nav#4255 · 1 条评论 · 1 个 reaction ·
维护者通常 3 天内回复
-
namespace operations
难度 1/5 1 小时以内 新手友好度 70/100
EclipseFdn/open-vsx.org#14421 ·
维护者通常 1 天内回复
-
untriaged
难度 2/5 1-3 小时 新手友好度 78/100
ubuntu/ubuntu-project-docs#850 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 83/100
维护者通常 1 天内回复