Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

LiteLLMChatTarget does not flag or survive output-token truncation

Aperta Adatta ai principianti
  • Qualcosa non va?
#2,878 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
84/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend-api-design

Direzione di ricerca

Inizia in pyrit/prompt_target/litellm_chat_target.py e confronta la gestione delle relative risposte con pyrit/prompt_target/openai/_response_adapter.py e openai_chat_target.py. Esegui tests/unit/prompt_target/target/test_litellm_chat_target.py, coprendo entrambi i casi riprodotti: una risposta con finish_reason="length" viene contrassegnata come troncata e una risposta troncata vuota restituisce un elemento vuoto senza sollevare un'eccezione.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

LiteLLMChatTarget does not handle output-token truncation, even though it speaks the same OpenAI-compatible Chat Completions shape that OpenAIChatTarget already handles.

With max_tokens set (pyrit/prompt_target/litellm_chat_target.py:138), a response that stops at the limit arrives with finish_reason == "length", and two things go wrong:

  1. A clipped answer is stored as a complete one. MessagePiece.mark_as_truncated exists for exactly this (pyrit/models/messages/message_piece.py:196): "A truncated piece may still carry a partial answer with response_error == "none", so without this flag a consumer cannot tell a complete answer from a clipped one." is_truncated stays False for a LiteLLMChatTarget piece, so a scorer or a red teamer reads a cut-off answer as the model's real answer.
  2. A truncation that empties the response aborts the turn. validate_chat_completion_response raises EmptyResponseException when there is no content, audio or tool call (pyrit/prompt_target/common/chat_completions_response_parser.py:107), and it runs before the construct step. This target is deliberately not wrapped in pyrit_target_retry (LiteLLM owns retry via num_retries), so nothing catches it — the conversation just stops.
How OpenAIChatTarget already handles the same case

pyrit/prompt_target/openai/openai_chat_target.py:314-331 warns, flags the first piece, and returns a graceful empty piece when truncation left nothing. pyrit/prompt_target/openai/_response_adapter.py:105-114 is the shape to mirror: validate() warns and returns early when is_truncated, otherwise calls validate_chat_completion_response; is_truncated() is get_finish_reason(response=response) == "length".

warn_truncated_response documents that its wording lives in common "to keep targets from drifting apart" (pyrit/prompt_target/common/utils.py:197), and build_empty_truncated_response is already shared for this case.

Steps/Code to Reproduce

Reproduced on main at 7b533109, through the public entry point (tests/unit/prompt_target/target/test_litellm_chat_target.py, litellm stubbed so no provider call is made). Two tests, one per failure mode:

truncated = _mock_response(content="The answer is", finish_reason="length")
litellm_stub.acompletion = AsyncMock(return_value=truncated)
result = await target.send_prompt_async(message=_user_message())
assert result[0].message_pieces[0].is_truncated is True          # currently False

empty = _mock_response(content=None, finish_reason="length")
empty.choices[0].message.content = None
litellm_stub.acompletion = AsyncMock(return_value=empty)
result = await target.send_prompt_async(message=_user_message())  # currently raises
assert result[0].message_pieces[0].response_error == "empty"
Expected Results

The first assertion holds; the second call returns a graceful empty piece instead of raising.

Actual Results
FAILED test_token_limit_truncation_marks_the_piece
E   AssertionError: assert False is True
     ... 'finish_reason': 'length'}).is_truncated

FAILED test_token_limit_truncation_with_no_content_does_not_raise
E   pyrit.exceptions.exception_classes.EmptyResponseException: Status Code: 204,
    Message: The chat returned an empty response (no content, audio, or tool_calls).
pyrit/prompt_target/common/chat_completions_response_parser.py:109: EmptyResponseException
Screenshots

Not applicable.

Versions
  • OS: macOS
  • Python version: 3.11+
  • PyRIT version: main at 7b533109
Proposed change

Confined to pyrit/prompt_target/litellm_chat_target.py: add _is_truncated_response on the existing get_finish_reason helper; warn via warn_truncated_response(..., limit_parameter="max_tokens") and skip strict validation when truncated; in _construct_message_from_response_async, mark_as_truncated() the first piece and return build_empty_truncated_response(...) when truncation left no pieces.

One question, since this changes error behaviour: should a truncated-but-empty LiteLLM response continue as error="empty" like the OpenAI target, or is aborting the turn preferred here? Happy to follow whichever the project prefers. A branch with the change and both tests is on feiiiiii5:fix/litellm-truncation-marker if you want to look at the diff.

Lingua principale
Python
Stelle
4.5k
Fork
896
Merge medio
3g 1h
PR unite (30g)
208

Preparare l'ambiente

Non abbiamo ancora controllato i file di configurazione di questo progetto. Parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/PyRIT

Tutte le issue di microsoft/PyRIT

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.