Crescendo backtracking keeps the refused prompt when the objective target made a tool call

Geschlossen
  • Stimmt etwas nicht?
#3,252 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 2 Tagen

Ein zugehöriger Pull Request wurde bereits gemerged.

  • #3253 von @RachelWanggg — gemerged

Bewertung

Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Anfängerfreundlichkeit
30/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
ai, databases

Rechercherichtung

The fixed-count slice is in duplicate_conversation_excluding_last_turn_async in pyrit/memory/memory_interface.py, and its only caller is pyrit/executor/attack/multi_turn/crescendo.py. Run the reproduction script in the issue to see the backtracked conversation keep question 2 and its function_call. Pull request #3253 is already open against this, so read it first. Done means a backtrack removes every message of the refused turn, including tool-call and tool-output messages, and leaves no orphan function_call.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Describe the bug

When CrescendoAttack backtracks after a refusal, duplicate_conversation_excluding_last_turn_async is supposed to drop the last turn, "before the last user request" per its docstring. It instead drops a fixed number of sequences: 1 if the last message is user or system, otherwise 2. That assumes a turn is one user message plus one assistant message.

An OpenAIResponseTarget with tools (custom_functions, tools or tool_providers) returns several messages per turn: the function call, the tool output and the final answer, and PromptNormalizer stores each one as its own sequence. If the refused turn made a tool call, the backtracked conversation keeps the refused prompt and that turn's function call, and drops only the tool output and the refusal. The next turn then sends the refused prompt again, plus a function_call with no matching function_call_output.

This is the only caller of the method in the library (crescendo.py).

Steps/Code to Reproduce
import asyncio
import json

from pyrit.memory import CentralMemory
from pyrit.models import MessagePiece
from pyrit.setup import IN_MEMORY, initialize_pyrit_async

CONVERSATION_ID = "crescendo-with-tools"


def tool_calling_turn(sequence: int, prompt: str, call_id: str, answer: str) -> list[MessagePiece]:
    # What OpenAIResponseTarget's tool loop stores for one turn: the prompt, the model's function_call,
    # the tool output and the final answer, each as its own message.
    return [
        MessagePiece(role="user", original_value=prompt, conversation_id=CONVERSATION_ID, sequence=sequence),
        MessagePiece(
            role="assistant",
            original_value=json.dumps({"type": "function_call", "call_id": call_id, "name": "lookup", "arguments": "{}"}),
            original_value_data_type="function_call",
            conversation_id=CONVERSATION_ID,
            sequence=sequence + 1,
        ),
        MessagePiece(
            role="tool",
            original_value=json.dumps({"type": "function_call_output", "call_id": call_id, "output": "result"}),
            original_value_data_type="function_call_output",
            conversation_id=CONVERSATION_ID,
            sequence=sequence + 2,
        ),
        MessagePiece(role="assistant", original_value=answer, conversation_id=CONVERSATION_ID, sequence=sequence + 3),
    ]


async def main() -> None:
    await initialize_pyrit_async(memory_db_type=IN_MEMORY)
    memory = CentralMemory.get_memory_instance()
    pieces = tool_calling_turn(0, "question 1", "call_1", "answer 1")
    pieces += tool_calling_turn(4, "question 2", "call_2", "I can't help with that")
    await memory.add_message_pieces_to_memory_async(message_pieces=pieces)

    # What CrescendoAttack does after the refusal scorer flags "question 2"
    backtracked_id = await memory.duplicate_conversation_excluding_last_turn_async(conversation_id=CONVERSATION_ID)
    for piece in await memory.get_message_pieces_async(conversation_id=backtracked_id):
        print(piece.api_role, piece.original_value_data_type, piece.original_value)


if __name__ == "__main__":
    asyncio.run(main())
Expected Results

Only the first turn is kept:

user text question 1
assistant function_call {"type": "function_call", "call_id": "call_1", "name": "lookup", "arguments": "{}"}
tool function_call_output {"type": "function_call_output", "call_id": "call_1", "output": "result"}
assistant text answer 1
Actual Results

The refused prompt and its function call are kept:

user text question 1
assistant function_call {"type": "function_call", "call_id": "call_1", "name": "lookup", "arguments": "{}"}
tool function_call_output {"type": "function_call_output", "call_id": "call_1", "output": "result"}
assistant text answer 1
user text question 2
assistant function_call {"type": "function_call", "call_id": "call_2", "name": "lookup", "arguments": "{}"}

With a real OpenAIResponseTarget (only the OpenAI client mocked) and PromptNormalizer, the request sent after the backtrack still contains the refused question, and its input has function_call ids ['call_1', 'call_2'] but function_call_output ids only ['call_1'].

Versions
  • OS: macOS (Apple Silicon)
  • Python version: 3.12.13
  • PyRIT version: installed from main (68487b2e) in editable mode
Vorherrschende Sprache
Python
Sterne
4.7k
Forks
1k
Ø Merge
2 T. 3 Std.
Gemergte PRs (30 T.)
450

Entwicklungsumgebung

In Codespaces öffnen

Startet den Dev-Container des Projekts im Browser, mit Ihrem eigenen GitHub-Konto.

  • Kein Dockerfile und keine Docker-Compose-Datei
  • Hat eine Pull-Request-Vorlage
  • Kein Beitragsleitfaden

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus microsoft/PyRIT

Alle Issues in microsoft/PyRIT

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.