DOC ConversationScorer comments say tool output is excluded from the scored text; the code includes it

Read pyrit/score/conversation_scorer.py and pyrit/score/scorer_prompt_validator.py, then run the neighboring test in tests/unit/score/test_conversation_history_scorer.py with a tool-role piece added. Check with maintainers whether tool output should be scored, and align the implementation, comments, and regression coverage with that decision.

Describe the bug

ConversationScorer builds the text that a harm/objective scorer judges, and three consecutive
comments in that function say raw tool output is deliberately left out of it — but the code includes
it, so a role="tool" piece is folded into the scored conversation and rendered as "Tool: …".

pyrit/score/conversation_scorer.py on main:

# Goes through each message in the conversation and appends user/assistant messages only
# Explicitly excludes system, tool, developer messages from being scored/included in conversation history
# they are allowed in validation but not included in the scored conversation text
for conv_message in conversation:
    for piece in conv_message.message_pieces:
        # Only include user and assistant messages in the conversation text
        if piece.api_role in ["user", "assistant", "tool"] and self._validator.is_role_supported(piece):

The validator does not close the gap: ScorerPromptValidator's supported_roles "Defaults to every
role except simulated_assistant" (pyrit/score/scorer_prompt_validator.py:45-47), so under the
default validator a tool piece passes both conditions.

Steps/Code to Reproduce

Offline unit-style reproduction (no keys, no network), following
tests/unit/score/test_conversation_history_scorer.py::test_conversation_history_scorer_filters_roles_correctly
and printing what the wrapped scorer actually receives. Pieces stored for one conversation id:
user, tool, developer, system, assistant, each with a distinct sentinel string; then
capture mock_scorer._score_nested_async.call_args.kwargs["scorable"].value.

Measured on microsoft/PyRIT main @ 2215c5b (Python 3.14.5, uv sync default groups):

SCORED TEXT repr:
'User: USER_SENTINEL\nTool: TOOL_SENTINEL\nAssistant: ASSISTANT_SENTINEL\n'
USER_SENTINEL      in scored text -> True
TOOL_SENTINEL      in scored text -> True
DEV_SENTINEL       in scored text -> False
SYS_SENTINEL       in scored text -> False
ASSISTANT_SENTINEL in scored text -> True

The neighbouring test already pins the intended shape for two of the roles — it asserts
expected_conversation = "User: User message\nAssistant: Assistant message\n" and
assert "System message" not in called_scorable.value — and it does not cover a tool piece at all,
which is why the divergence is invisible to CI.

Expected Results

One of the two, and this is the question I cannot answer from the code alone:

• either tool output is excluded, matching the three comments (drop "tool" from the role list), or
• tool output is included on purpose, in which case the comments are wrong and a test should pin the
  behaviour so the two cannot drift again.

Actual Results

Tool text is included in the conversation the scorer judges, while developer and system are not.

Why I think it matters for scoring correctness

A scorer is being asked to judge what the target produced. Third-party tool results are neither a
user turn nor an assistant turn, so a long or hostile tool payload sitting in the judged text can move
the verdict in either direction without reflecting the target's own behaviour. scorer_prompt_validator.py:9-14
gives the analogous reason for dropping simulated history — a scorer that judges the target must not
mistake it for what the target said.

The history explains the split rather than settling it: the comments came in with the original
ConversationScorer (#1138, 2337fce7, 2025-12-12), while "tool" was added to the whitelist by
#2518 "Policy Scorer Compatibility (phase 2.5)" (8b3826be, 2026-09-02), which left the adjacent
comments untouched. So I cannot tell whether #2518 intended policy scorers to see tool output and the
comments simply went stale, or whether the widening was incidental.

If tool visibility is required only for the policy-scorer path, a validator-driven option is to keep
the exclusion in ConversationScorer and let a scorer that wants tool output declare it through
supported_roles, so the include/exclude decision lives in one place instead of two.

Not verified, and I would not want a fix to assume it: I have not checked whether any shipped scenario
actually scores a conversation that contains tool pieces, so I cannot say how often this changes a real
verdict in practice — only that the default path includes them.

Versions

• OS: macOS 27.0 (arm64)
• Python: 3.14.5 (also reproduced logic review on 3.11.15)
• PyRIT: 1.2.0.dev0 from main @ 2215c5b