AI providers: tool results cannot carry images because non-string results are JSON.stringify'd

クローズ
  • 何か問題がありますか？
#9,857 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@nmnmcc がすでに取り組んでいます。

2026年10月23日 から。

  • #9860 @nmnmcc による — オープン
  • #9861 @effect-bot による — マージ済み

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
typescript
領域
ai, backend-api-design

調査の方向性

Read packages/effect/src/ai/Toolkit.ts and Prompt.ts, then trace tool result serialization in packages/ai/anthropic/src/AnthropicLanguageModel.ts, packages/ai/openai/src/OpenAiLanguageModel.ts, and packages/ai/openai-compat/src/OpenAiLanguageModel.ts. Compare the Anthropic and OpenAI Responses content formats described in the issue and decide how model-facing content should be represented. Done means supported rich results reach compatible APIs in their native form, with text-only providers retaining current behavior or reporting a clear error.

索引モデルが issue の本文から書いたものです。

説明

What is the problem?

A tool result cannot carry an image (or other content blocks) to the model. The provider packages turn every non-string tool result into a string with JSON.stringify, so the model gets the image as plain text.

Both provider APIs support rich tool results:

  • Anthropic Messages: tool_result.content can be a string or an array of content blocks, including image (base64 or URL) and document blocks.
  • OpenAI Responses: function_call_output.output can be a string or an array of input_text, input_image, and input_file items.

Where it happens

The path from a tool handler to the request (on main, 4d24a11; the same in 4.0.5 and 4.0.6):

  1. Toolkit encodes the handler result with the tool's success schema. There is no hook to map it to model content:
    https://github.com/Effect-TS/effect/blob/4d24a11217edda68e8c4c72cdc1cff025afa19d6/packages/effect/src/ai/Toolkit.ts#L527
  2. Prompt.ToolResultPart.result is unknown, and it gets encodedResult:
    https://github.com/Effect-TS/effect/blob/4d24a11217edda68e8c4c72cdc1cff025afa19d6/packages/effect/src/ai/Prompt.ts#L2268
  3. The providers stringify anything that is not a string: 
content: typeof part.result === "string" ? part.result : JSON.stringify(part.result),

Example

A tool (for example, a screenshot tool) returns this success value:

[{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw0..." } }]

@effect/ai-anthropic sends:

{ "type": "tool_result", "tool_use_id": "...", "content": "[{\"type\":\"image\",\"source\":{...}}]" }

The API treats the string as one text block. So:

  • The model cannot see the image. It only reads base64 characters.
  • The base64 text uses a very large number of tokens. A 500 KB PNG is about 670,000 characters, which can be more than the whole context window. A real image block costs about width * height / 750 tokens on Claude.

The same thing happens with function_call_output.output on OpenAI.

Expected behavior

A tool should be able to return model-facing content parts (text, image, file), and each provider should send them in its native form:

  • Anthropic: tool_result.content as an array of text / image / document blocks.
  • OpenAI Responses: function_call_output.output as an array of input_text / input_image / input_file.
  • Providers or APIs that only accept text tool results (for example, Chat Completions tool messages) can keep the current string behavior, or fail with a clear error.

Possible API

Two ideas (I am open to others):

  1. An optional mapping on the tool, like toModelOutput in the Vercel AI SDK. It maps the decoded result to content parts that use the existing Prompt part types (TextPart, FilePart). The result that the program sees stays typed as today.
  2. A known shape for ToolResultPart.result, such as { type: "content", value: Array<TextPart | FilePart> }, that providers check for before they fall back to JSON.stringify.

Workaround today

Put the image in a separate user message after the tool result. This works, but it moves the image out of the tool call that made it, and it does not fit well with LanguageModel.generateText tool loops.

Versions

The problem is in 4.0.5 and in the latest release, 4.0.6. The reproduction in the comment below gives the same output on both.

  • effect 4.0.5, 4.0.6
  • @effect/ai-anthropic 4.0.5, 4.0.6
  • @effect/ai-openai 4.0.5, 4.0.6
  • @effect/ai-openai-compat 4.0.5, 4.0.6
主要言語
TypeScript
スター
16.7k
フォーク
808
平均マージ
11時間 3分
マージ済み PR（30日）
840

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

Effect-TS/effect のほかの issue

Effect-TS/effect の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。