Proposal: DecayDeltaScorer — a turn-over-turn rate-of-change scorer for multi-turn attacks

Bắt đầu bằng cách đọc memory_interface.py, đặc biệt là get_scores() và get_conversation_messages(), sau đó so sánh các mẫu wrapping của FloatScaleScorer, MessageFloatScaleScorer và FloatScaleThresholdScorer. Xác định xem các lượt trước có nên được chấm điểm lại hay các score đã được lưu bền vững nên được truy xuất theo ID của từng message-piece, đồng thời xác định hành vi mong đợi đối với các delta giữa những lượt liên tiếp trước khi triển khai.

Motivation

PyRIT has strong multi-turn attack orchestration (including Crescendo) and a ConversationScorer that evaluates conversation context by re-scoring the accumulated history. However, I haven't found a scorer that explicitly captures the rate of risk change between turns without repeatedly evaluating the full conversation.

I also searched the repository for related concepts such as delta, rate_of_change, turning_point, and decay across pyrit/score and pyrit/executor, but didn't find an existing equivalent. Happy to be pointed to an existing implementation if I've missed one.

Proposed approach

I'd like to propose a DecayDeltaScorer, implementing FloatScaleScorer / MessageFloatScaleScorer and following the wrapping pattern used by FloatScaleThresholdScorer.

Rather than thresholding the wrapped scorer's output, it would calculate the change in score between consecutive turns:

Δ(t) = S(t) − S(t−1)

where S(t) is the wrapped scorer's score for the current turn and S(t−1) is its score for the previous turn in the same conversation.

The goal is for this to be a deterministic, lightweight complement to ConversationScorer, rather than a replacement:

• ConversationScorer: "How risky is the conversation at this point?"
• DecayDeltaScorer: "How quickly is the risk changing?"

This could be particularly useful for gradual, multi-turn escalation such as Crescendo attacks, where individual turns may remain below a detection threshold while the overall trajectory is consistently increasing.

Implementation question

This is the main reason I'm opening an issue before submitting a PR.

I traced get_scores() in memory_interface.py. It supports filtering by score type/category/timestamp/scorer identifier, but I don't see a direct conversation-level filter for retrieving the wrapped scorer's previous score.

Two possible approaches I see are:

1. Re-score previous turns retrieved through get_conversation_messages().
2. Retrieve persisted scores using the message-piece IDs associated with the conversation.

I'd appreciate guidance on which approach better fits PyRIT's existing architecture and performance expectations before I proceed with an implementation.

Background

This proposal is motivated by my research into trajectory-level detection of gradual multi-turn attacks. In particular, I've observed cases where per-turn monitoring provides little or no pre-critical signal, while monitoring the trajectory reveals a consistent increase in risk.

I'm happy to share additional methodology or benchmark details if they would be useful for evaluating the proposed scoring approach.

If the approach fits PyRIT's scoring architecture, I'd be happy to follow up with a PR.