Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

LiteLLMChatTarget does not flag or survive output-token truncation

未关闭 适合新手
  • 哪里不对？
#2,878 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
84/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
python
领域
backend-api-design

调研方向

从 pyrit/prompt_target/litellm_chat_target.py 开始，将其响应处理与 pyrit/prompt_target/openai/_response_adapter.py 和 openai_chat_target.py 进行比较。运行 tests/unit/prompt_target/target/test_litellm_chat_target.py，覆盖已复现的两种情况：带有 finish_reason="length" 的响应会被标记为已截断，而空的截断响应会返回一个空元素且不会引发异常。

由索引模型根据 Issue 内容生成。

描述

Describe the bug

LiteLLMChatTarget does not handle output-token truncation, even though it speaks the same OpenAI-compatible Chat Completions shape that OpenAIChatTarget already handles.

With max_tokens set (pyrit/prompt_target/litellm_chat_target.py:138), a response that stops at the limit arrives with finish_reason == "length", and two things go wrong:

  1. A clipped answer is stored as a complete one. MessagePiece.mark_as_truncated exists for exactly this (pyrit/models/messages/message_piece.py:196): "A truncated piece may still carry a partial answer with response_error == "none", so without this flag a consumer cannot tell a complete answer from a clipped one." is_truncated stays False for a LiteLLMChatTarget piece, so a scorer or a red teamer reads a cut-off answer as the model's real answer.
  2. A truncation that empties the response aborts the turn. validate_chat_completion_response raises EmptyResponseException when there is no content, audio or tool call (pyrit/prompt_target/common/chat_completions_response_parser.py:107), and it runs before the construct step. This target is deliberately not wrapped in pyrit_target_retry (LiteLLM owns retry via num_retries), so nothing catches it — the conversation just stops.
How OpenAIChatTarget already handles the same case

pyrit/prompt_target/openai/openai_chat_target.py:314-331 warns, flags the first piece, and returns a graceful empty piece when truncation left nothing. pyrit/prompt_target/openai/_response_adapter.py:105-114 is the shape to mirror: validate() warns and returns early when is_truncated, otherwise calls validate_chat_completion_response; is_truncated() is get_finish_reason(response=response) == "length".

warn_truncated_response documents that its wording lives in common "to keep targets from drifting apart" (pyrit/prompt_target/common/utils.py:197), and build_empty_truncated_response is already shared for this case.

Steps/Code to Reproduce

Reproduced on main at 7b533109, through the public entry point (tests/unit/prompt_target/target/test_litellm_chat_target.py, litellm stubbed so no provider call is made). Two tests, one per failure mode:

truncated = _mock_response(content="The answer is", finish_reason="length")
litellm_stub.acompletion = AsyncMock(return_value=truncated)
result = await target.send_prompt_async(message=_user_message())
assert result[0].message_pieces[0].is_truncated is True          # currently False

empty = _mock_response(content=None, finish_reason="length")
empty.choices[0].message.content = None
litellm_stub.acompletion = AsyncMock(return_value=empty)
result = await target.send_prompt_async(message=_user_message())  # currently raises
assert result[0].message_pieces[0].response_error == "empty"
Expected Results

The first assertion holds; the second call returns a graceful empty piece instead of raising.

Actual Results
FAILED test_token_limit_truncation_marks_the_piece
E   AssertionError: assert False is True
     ... 'finish_reason': 'length'}).is_truncated

FAILED test_token_limit_truncation_with_no_content_does_not_raise
E   pyrit.exceptions.exception_classes.EmptyResponseException: Status Code: 204,
    Message: The chat returned an empty response (no content, audio, or tool_calls).
pyrit/prompt_target/common/chat_completions_response_parser.py:109: EmptyResponseException
Screenshots

Not applicable.

Versions
  • OS: macOS
  • Python version: 3.11+
  • PyRIT version: main at 7b533109
Proposed change

Confined to pyrit/prompt_target/litellm_chat_target.py: add _is_truncated_response on the existing get_finish_reason helper; warn via warn_truncated_response(..., limit_parameter="max_tokens") and skip strict validation when truncated; in _construct_message_from_response_async, mark_as_truncated() the first piece and return build_empty_truncated_response(...) when truncation left no pieces.

One question, since this changes error behaviour: should a truncated-but-empty LiteLLM response continue as error="empty" like the OpenAI target, or is aborting the turn preferred here? Happy to follow whichever the project prefers. A branch with the change and both tests is on feiiiiii5:fix/litellm-truncation-marker if you want to look at the diff.

主要语言
Python
星标
4.5k
派生
896
平均合并
3 天 1 小时
30 天内合并 PR
208

环境准备

我们还没有检查这个项目的环境配置文件。先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

microsoft/PyRIT 的其他 Issue

查看 microsoft/PyRIT 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。