Bound and instrument large plugin tool-result payloads in chat turns

Summary

Plugin (action) tool results are retained and passed around a chat turn without a shared size bound, and we have no diagnostics that would let us distinguish a memory problem from a Cosmos write problem from a model context-window problem.

A customer running Simple Chat locally reports the app crashing when an action returning large result sets is combined with an Azure AI Search workspace. We cannot currently diagnose that from logs, because nothing reports how much data a turn actually accumulated or where it went.

Add size instrumentation first so the failure mode can be identified, then apply a shared bounding policy for plugin results. This is not specific to any one action type.

User Value

• Admins and developers can see why a chat turn failed instead of getting an opaque crash or a generic error.
• Actions that legitimately return large datasets degrade predictably instead of taking down the process.
• Removes the need for each action type to invent its own limits, which is the current situation and is inconsistent.

Proposed Behavior

Phase 1 — Diagnostics

Instrument with debug_print / log_event:

• Serialized byte size of each plugin invocation result, attributed to plugin and function name.
• Cumulative tool-result bytes accumulated for the current turn.
• Total retained size of PluginInvocationLogger.invocations after each append, not just the item count.
• Assembled prompt payload size immediately before the model call, broken down by conversation history, document/search citations, and tool results.
• Explicit classification of Azure OpenAI 400 responses, distinguishing context_length_exceeded from other bad requests.

Phase 2 — Shared bounding policy

• A single configurable cap for plugin tool-result payloads, applied consistently rather than per action type.
• Bound what is retained in memory, not only what is logged or persisted.
• Surface truncation explicitly to the model and to the user, so a truncated result is never silently presented as complete.

Acceptance Criteria

• Per-invocation and per-turn tool-result sizes appear in debug logging with plugin and function attribution.
• Retained invocation-history size is observable and bounded by bytes, not only by item count.
• Prompt assembly logs a size breakdown before the model call.
• Azure OpenAI context-length errors are logged distinctly and surfaced as an actionable message rather than a generic failure.
• A shared plugin result cap exists and applies regardless of action type.
• Truncation is explicitly marked in the result handed to the model.
• Functional test covering an oversized plugin result, asserting bounding and truncation signaling.

Notes

Current state

• PluginInvocationLogger.__init__ (semantic_kernel_plugins/plugin_invocation_logger.py L343) retains max_history = 1000 invocations holding the full, untruncated result object. Truncation via sanitize_plugin_invocation_value (max_string_length=500) and MAX_SAFE_INVOCATION_STRING_LENGTH = 20000 is applied to logging and persistence output only, never to retained state.
• get_invocations_for_conversation(..., limit=1000) is called from at least eight sites in route_backend_chats.py during a single turn, including L3000, L3024, L18511, L18874, and L19418.
• Per-action-type limits exist and are inconsistent: MCP_MAX_TOOL_RESULT_TEXT_LENGTH = 120000 in functions_mcp_operations.py; the Yamcs action defaults to max_rows = 500 and byte_limit = 250000; several other action types have no cap at all.
• Search contributes independently: SEARCH_DEFAULT_TOP_N = 12 and SEARCH_MAX_TOP_N = 500 in functions_search.py.
• conversation_history_limit (default 10) replays prior messages, which may already carry large tool payloads from earlier turns.

Pattern to reuse

build_evidence_envelope in functions_mixed_source_orchestration.py L1121 already implements bounded JSON lists and UTF-8-safe truncation with an explicit bounds_applied flag. That is the shape the shared policy should follow rather than inventing a new one.

Leading hypothesis

Process memory exhaustion from retained full-size invocation results, not model context overflow. A context overflow would return a clean Azure OpenAI 400, which the customer is not seeing. Retaining up to 1000 full results and re-reading them from many call sites per turn is the mechanism that scales badly. This is masked in App Service by gunicorn worker recycling and available RAM, and manifests as a hard process exit on a developer workstation running other workloads alongside it.

Secondary candidates to confirm or eliminate with the same instrumentation, rather than assume:

• Cosmos DB 2 MB item size limit on message save when citations carry large tool results. The log_tool_execution path in agent_logging_chat_completion.py L39 truncates function_result to 500 characters, but citations built from the invocation logger take a different path that needs verifying.
• Repeated deep copy and serialization of large results across the sanitize → citation → persist chain.
• The Werkzeug development server with threaded=True being less resilient under memory pressure than gunicorn.

Related gap

route_backend_chats.py L14291 is the only place that pattern-matches BadRequestError, and it does not distinguish context-length failures. If the model is reporting an oversized prompt, we are currently discarding that signal.

Staging

Phase 1 is independently valuable and should ship first. Phase 2 should not be scoped until Phase 1 data identifies the actual failure mode.