Structured Outputs not supporting meta-llama/Llama-3.2-3B-Instruct-Turbo

Open
#173 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
typescript
Domain
api

Research direction

Reproduce the failure with together.chat.completions.create using meta-llama/Llama-3.2-3B-Instruct-Turbo and the shown response_format, then compare it with Meta-Llama-3.1-8B-Instruct-Turbo. Trace the Structured Outputs request path in the TypeScript client and verify whether the model is actually accepted; done means the documented model accepts the same schema without the grammar validation error.

Written by the indexing model from the issue text.

Description

Even thought it's stated that meta-llama/Llama-3.2-3B-Instruct-Turbo is supported for Structured Outputs, this code returns the following error:

 const extract = await together.chat.completions.create({
    messages: [
      {
        role: "system",
        content:
          "The following is a list of items from a menu. Only answer in JSON.",
      },
      {
        role: "user",
        content: menuItems!,
      },
    ],
    model: "meta-llama/Llama-3.2-3B-Instruct-Turbo",
    // @ts-expect-error - this is not typed in the API
    response_format: { type: "json_object", schema: jsonSchema },
  });

Error:

error: {
    id: 'npsvkwr-57nCBj-92f800485c83c4bf',
    error: {
      message: "Input validation error: grammar is not valid: grammar is not valid: Grammar must have a 'properties' field",
      type: 'invalid_request_error',
      param: null,
      code: null
    }
  }
}

The same code works with meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo.

Dominant language
TypeScript
Stars
71
Forks
10
Avg merge
3d 10h
Merged PRs (30d)
5

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.