evals run_inference (local ADK agent): strips multimodal prompt parts (inline_data) before Runner.run_async

Environment

• google-cloud-aiplatform 1.157.0
• google-agents-cli 0.4.0 (agents-cli eval generate)
• google-adk 2.2.0

Summary

vertexai.Client().evals.run_inference(agent=<local_adk_agent>) — the path used by agents-cli eval generate — drops non-text prompt parts before calling ADK Runner.run_async. Eval cases with multimodal prompt.parts[] (e.g. PDF inline_data per agents-cli multimodal eval schema) reach the agent as text-only user content. Tools that depend on tool_context.user_content.parts[].inline_data never see the attachment.

Repro

1. Create a one-case eval dataset with multimodal prompt (text + PDF inline_data), e.g. agents-cli multimodal schema:

{
  "eval_cases": [{
    "eval_case_id": "pdf_smoke",
    "prompt": {
      "role": "user",
      "parts": [
        {"text": "Process this document."},
        {"inline_data": {"mime_type": "application/pdf", "data": "<base64>"}}
      ]
    }
  }]
}

2. Run local inference via agents-cli:

agents-cli eval generate --dataset <dataset.json> --output traces.json

3. Inspect trace: agent tool that reads user_content.parts for inline_data returns an error like "No document found" — only the text part survived inference.

Contrast (works)

ADK AgentEvaluator / pytest evaluation passes the full Content (text + inline_data) into Runner.run_async, and the same tool succeeds.

Root cause

In vertexai/_genai/_evals_common.py:

1. DataFrame build flattens prompt to text

_eval_cases_to_dataframe stores:

row[PROMPT] = _evals_data_converters._get_content_text(case.prompt)

_get_content_text (_evals_data_converters.py) concatenates part.text and ignores inline_data, file_data, etc.

2. Local agent run wraps string as single text Part

_execute_local_agent_run_with_retry_async calls:

new_message_content = genai_types.Content(
    role="user",
    parts=[genai_types.Part(text=contents)],
)

So even if contents were a full Content, this stringifies it into one text part. Today contents is already the plain text string.

3. N+1 / agent_data path — same bug

When inference uses agent_data turns, the code still calls _get_content_text(last_user_content) before the local agent run.

Expected behavior

Preserve full multimodal Content (all parts, including inline_data / file_data) through the local-agent inference path and call:

Runner.run_async(new_message=Content.model_validate(full_content))

instead of Part(text=...).

Suggested fix locations

vertexai/_genai/_evals_common.py:

1. _eval_cases_to_dataframe — store full prompt Content dict (with parts) when multimodal, not _get_content_text() only.
2. _extract_contents_for_inference — if dict has parts, pass through (do not require nested contents key).
3. _execute_local_agent_run_with_retry_async — use Content.model_validate(contents) when contents is a Content dict; keep Part(text=...) fallback for plain strings.
4. N+1 agent_data branch — same helper instead of _get_content_text(last_user_content).

Impact

Blocks multimodal agent eval via agents-cli eval generate for any tool that reads uploaded bytes from user_content (PDF invoices, images, etc.). Workaround today is ADK AgentEvaluator pytest path, which preserves inline_data.

References

• agents-cli multimodal eval schema: https://github.com/google/agents-cli/blob/main/skills/google-agents-cli-eval/references/multimodal-eval.md
• Related open eval issues: #6785