run-llama/LlamaIndexTS

add detail to image_url for openai chat completions

Aperta

#2221 aperta il 20 ott 2025

 (3 commenti) (0 reazioni) (0 assegnatari)TypeScript (524 fork)auto 404
buggood first issuehelp wanted

Metriche repository

Star
 (3078 stelle)
Metriche merge PR
 (Nessuna PR mergiata in 30 g)

Descrizione

Describe the bug When sending messages containing image content to OpenAI's chat completions endpoint, the detail is ignored. OpenAI expects it to be inside the image_url object (see here), but llamaindex expects it to be next to the image_url: https://github.com/run-llama/LlamaIndexTS/blob/fc385dc16757a73dc19ca239ebbe1e6155ed4b4d/packages/core/src/llms/type.ts#L195-L199

and there's no translation happening when using llm.chat: https://github.com/run-llama/LlamaIndexTS/blob/fc385dc16757a73dc19ca239ebbe1e6155ed4b4d/packages/providers/openai/src/llm.ts#L248-L249

To Reproduce Code to reproduce the behavior:

import { OpenAI, OpenAIResponses } from "@llamaindex/openai";
import type { ChatMessage } from "llamaindex";

const followingTypes: ChatMessage = {
  role: "user",
  content: [
    {
      type: "image_url",
      detail: "high",
      image_url: { url: "data:image/jpeg;base64,aGVsbG8=" },
    },
  ],
};

const workaroundForChatCompletionsOnly: ChatMessage = {
  role: "user",
  content: [
    {
      type: "image_url",
      image_url: {
        url: "data:image/jpeg;base64,aGVsbG8=",
        // @ts-expect-error
        detail: "high",
      },
    },
  ],
};

const workaroundForBoth: ChatMessage = {
  role: "user",
  content: [
    {
      type: "image_url",
      detail: "high",
      image_url: {
        url: "data:image/jpeg;base64,aGVsbG8=",
        // @ts-expect-error
        detail: "high",
      },
    },
  ],
};

const messages = [
  followingTypes,
  workaroundForChatCompletionsOnly,
  workaroundForBoth,
];

const chat = OpenAI.toOpenAIMessage(messages);
const responses = new OpenAIResponses().toOpenAIResponseMessages(messages);

console.dir({ chat, responses }, { depth: null });

The workaroundForBoth ends up putting detail in two places for chat completions. I still need to confirm if this throws a 4xx error from OpenAI or if it's just ignored.

{
  "type": "image_url",
  "detail": "high",
  "image_url": {
    "url": "data:image/jpeg;base64,aGVsbG8=",
    "detail": "high"
  }
} 

Expected behavior We should be able to provide messages that adhere to llamaindex's ChatMessage type and still have the detail param make it into the correct spot for both Chat Completions and Responses APIs.

i.e. OpenAI.toOpenAIMessage([followingTypes]) should return:

[
  {
    "role": "user",
    "content": [
      {
        "type": "image_url",
        "image_url": {
          "url": "data:image/jpeg;base64,aGVsbG8=",
          "detail": "high"
        }
      }
    ]
  }
]

And new OpenAIResponses().toOpenAIResponseMessages([followingTypes]) should return:

[
  {
    "role": "user",
    "content": [
      {
        "type": "input_image",
        "image_url": "data:image/jpeg;base64,aGVsbG8=",
        "detail": "high"
      }
    ]
  }
]

Screenshots N/A

Desktop (please complete the following information):

  • OS: macOS
  • JS Runtime / Framework / Bundler (select all applicable)
  • Node.js
  • Deno
  • Bun
  • Next.js
  • ESBuild
  • Rollup
  • Webpack
  • Turbopack
  • Vite
  • Waku
  • Edge Runtime
  • AWS Lambda
  • Cloudflare Worker
  • Others (please elaborate on this)
  • Version: 1.2.22

Additional context N/A

Guida contributor