Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

AI providers: tool results cannot carry images because non-string results are JSON.stringify'd

Open
  • Is something wrong?
#9,857 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@nmnmcc is already working on this.

Since Oct 23, 2026.

  • #9860 by @nmnmcc — open
  • #9861 by @effect-bot — open

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
typescript
Domain
ai, backend-api-design

Research direction

Read packages/effect/src/ai/Toolkit.ts and Prompt.ts, then trace tool result serialization in packages/ai/anthropic/src/AnthropicLanguageModel.ts, packages/ai/openai/src/OpenAiLanguageModel.ts, and packages/ai/openai-compat/src/OpenAiLanguageModel.ts. Compare the Anthropic and OpenAI Responses content formats described in the issue and decide how model-facing content should be represented. Done means supported rich results reach compatible APIs in their native form, with text-only providers retaining current behavior or reporting a clear error.

Written by the indexing model from the issue text.

Description

What is the problem?

A tool result cannot carry an image (or other content blocks) to the model. The provider packages turn every non-string tool result into a string with JSON.stringify, so the model gets the image as plain text.

Both provider APIs support rich tool results:

  • Anthropic Messages: tool_result.content can be a string or an array of content blocks, including image (base64 or URL) and document blocks.
  • OpenAI Responses: function_call_output.output can be a string or an array of input_text, input_image, and input_file items.

Where it happens

The path from a tool handler to the request (on main, 4d24a11; the same in 4.0.5 and 4.0.6):

  1. Toolkit encodes the handler result with the tool's success schema. There is no hook to map it to model content:
    https://github.com/Effect-TS/effect/blob/4d24a11217edda68e8c4c72cdc1cff025afa19d6/packages/effect/src/ai/Toolkit.ts#L527
  2. Prompt.ToolResultPart.result is unknown, and it gets encodedResult:
    https://github.com/Effect-TS/effect/blob/4d24a11217edda68e8c4c72cdc1cff025afa19d6/packages/effect/src/ai/Prompt.ts#L2268
  3. The providers stringify anything that is not a string: 
content: typeof part.result === "string" ? part.result : JSON.stringify(part.result),

Example

A tool (for example, a screenshot tool) returns this success value:

[{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw0..." } }]

@effect/ai-anthropic sends:

{ "type": "tool_result", "tool_use_id": "...", "content": "[{\"type\":\"image\",\"source\":{...}}]" }

The API treats the string as one text block. So:

  • The model cannot see the image. It only reads base64 characters.
  • The base64 text uses a very large number of tokens. A 500 KB PNG is about 670,000 characters, which can be more than the whole context window. A real image block costs about width * height / 750 tokens on Claude.

The same thing happens with function_call_output.output on OpenAI.

Expected behavior

A tool should be able to return model-facing content parts (text, image, file), and each provider should send them in its native form:

  • Anthropic: tool_result.content as an array of text / image / document blocks.
  • OpenAI Responses: function_call_output.output as an array of input_text / input_image / input_file.
  • Providers or APIs that only accept text tool results (for example, Chat Completions tool messages) can keep the current string behavior, or fail with a clear error.

Possible API

Two ideas (I am open to others):

  1. An optional mapping on the tool, like toModelOutput in the Vercel AI SDK. It maps the decoded result to content parts that use the existing Prompt part types (TextPart, FilePart). The result that the program sees stays typed as today.
  2. A known shape for ToolResultPart.result, such as { type: "content", value: Array<TextPart | FilePart> }, that providers check for before they fall back to JSON.stringify.

Workaround today

Put the image in a separate user message after the tool result. This works, but it moves the image out of the tool call that made it, and it does not fit well with LanguageModel.generateText tool loops.

Versions

The problem is in 4.0.5 and in the latest release, 4.0.6. The reproduction in the comment below gives the same output on both.

  • effect 4.0.5, 4.0.6
  • @effect/ai-anthropic 4.0.5, 4.0.6
  • @effect/ai-openai 4.0.5, 4.0.6
  • @effect/ai-openai-compat 4.0.5, 4.0.6
Dominant language
TypeScript
Stars
16.7k
Forks
808
Avg merge
11h 24m
Merged PRs (30d)
848

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Effect-TS/effect

All issues in Effect-TS/effect

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.