Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

MLXLanguageModel tool calling: flaky repeat check, missing assistant tool-call turn, and tool-round limits

Open
  • Is something wrong?
#328 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
18/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
macos, swift
Domain
api, backend

Research direction

Start in MLXLanguageModel.swift at the repeated-call check (around lines 1230 and 1496), where the key-order-dependent string is built, and compare it with ToolRoundLimit (#262). The issue also covers the missing assistant tool-call turn in pendingChat and the decodingFailure behaviour at the tool-round limit, which needs maintainer design decisions. Done would need the maintainers' answers to the three questions before any patch.

Written by the indexing model from the issue text.

Description

Version: 1.0.0 (80d50b0), mlx-swift-lm 3.32.3, MLX 4-bit models on an M3 Pro (macOS 27).

We measured how MLXLanguageModel handles multi-round tool calling with small local models, and found three things.
We'd like your view before going any further. Details are in the collapsed sections.

1. The repeated-call check depends on dictionary key order

MLXLanguageModel stops a response when a tool round repeats the previous round's calls. It compares strings built
with "\($0.function.name):\($0.function.arguments)", where arguments is a [String: MLXLMCommon.JSONValue]
(MLXLanguageModel.swift:1230-1233, :1496-1499).

A native Swift Dictionary seeds its hash table with its storage address (hashSeed(for:scale:) in the standard
library's HashTable.swift), so two identical calls can print their keys in different orders. As a result, the check
is reliable for tools with zero or one argument and a coin flip for tools with more.

Evidence

Controlled run.

  • Greedy sampling fixes the model's output, so between runs only Swift's hashing differs.
  • The prompts MLX builds are byte-identical with and without SWIFT_DETERMINISTIC_HASHING=1 (swift-jinja sorts keys).

Identical consecutive rounds, caught / missed:

model, tool (arguments) default SWIFT_DETERMINISTIC_HASHING=1
Qwen3-4B, check_status(job_id, region) (2) 4 / 16 20 / 0
Qwen3-1.7B, a search tool (pattern, file, context) (3) 19 / 12 20 / 0
Qwen3-4B, check_status(job_id) (1) 10 / 0 10 / 0

Comparing the arguments with sorted keys instead removes every miss (20 / 0 and 20 / 0), which confirms the cause.

What the check compared. We logged both strings just before the comparison. Here is one missed repeat
(Gemma 4 e2b):

previous: calculate:["a": MLXLMCommon.JSONValue.string("12"), "b": MLXLMCommon.JSONValue.string("7"), "operation": MLXLMCommon.JSONValue.string("multiply")]
current:  calculate:["b": MLXLMCommon.JSONValue.string("7"), "a": MLXLMCommon.JSONValue.string("12"), "operation": MLXLMCommon.JSONValue.string("multiply")]

Across 14 models and 603 identical consecutive rounds, the check missed 286 times. Every miss was the same text in a
different order.

Without MLX:

import Foundation
let text = Data(#"{"pattern":"report","file":"notes.md","context":"0"}"#.utf8)
let rounds = try (0..<100).map { _ in try JSONDecoder().decode([String: String].self, from: text) }
print(Set(rounds.map { "\($0)" }).count)  // 6 (20 of 20 runs); 1 with SWIFT_DETERMINISTIC_HASHING=1 or a single key

For comparison, ToolRoundLimit (#262) compares calls structurally for the remote providers, and
LlamaLanguageModel signs calls with .sortedKeys JSON.

2. The assistant turn that made a tool call is never sent back to the model

Between rounds, the chat is rebuilt from the transcript plus pendingChat, but the assistant turn carrying the calls
is never added:

  • pendingChat gets .assistant(text) only when the round produced text (:1217-1219, :1485-1487).
  • Results are appended as .tool(content), with no id and no name (:1267, :1525).
  • convertTranscriptToMLXChat drops .toolCalls entries (:1737-1739).

So the chat template receives [system, user, tool]: a result with no call before it. What happens next depends on
the template:

  • Gemma 4 (e2b, e4b, their QAT builds; the 26B-A4B QAT template behaves the same) drops the result.
    • The round-2 prompt is byte-identical to round 1, so the model repeats the same call until the repeat check or the
      8-round cap stops it.
    • Across 280 round transitions, the result never appeared in the prompt.
  • gpt-oss-20b's template raises Message has tool role, but there was no previous assistant message with a tool call!. We checked this at the template level only.
  • Ministral 3's template raises After the optional system message, conversation roles must alternate user and assistant roles except for tool calls and results. once two rounds in a response produced text (2 of 6 runs).
  • Qwen3, Qwen3.5, Llama 3.2 and LFM2 render the result, but without the call that produced it.
Evidence: the same build with only the assistant turn added to pendingChat

Default sampling, 3–5 runs per mode, respond and streamResponse:

model scenario 1.0.0 + assistant turn
Gemma 4 e2b calculator ("12 * 7 + 3") 6/6 failed 6/6 answered 87
Gemma 4 e2b search tool that never matches 6/6 failed 6/6 answered that nothing was found
Gemma 4 e4b calculator 6/6 failed 6/6 answered 87
Gemma 4 E2B QAT calculator 6/6 failed 5/6 answered 87
Gemma 4 E4B QAT calculator 6/6 failed 4/6 answered 87 (1 empty answer)
Ministral-3-3B calculator 2/6 raised the alternation error 0/6
Qwen3-1.7B search tool 10/10 re-sent their previous call verbatim 0/10
Qwen3-8B search tool 6/6 re-sent their previous call verbatim 0/6
Qwen3-4B search tool 8/10 ended by announcing another search ("I will attempt to find … by searching the knowledgebase files") 0/10

Gemma's prompt, decoded, for "What is 12 * 7 + 3?".

  • On 1.0.0: rounds 2–9 are identical to round 1 (144 tokens).
  • With the turn: round 2 is round 1 plus
    <|tool_call>call:calculate{a:<|"|>12<|"|>,b:<|"|>7<|"|>,operation:<|"|>multiply<|"|>}<tool_call|><|tool_response>response:{value:<|"|>"84"<|"|>}<tool_response|>,
    and the model's next call is add(84, 3).

Llama 3.x. Its templates reject an assistant message carrying more than one call, with This model only supports single tool-calls at once!. With all of a round's calls in one message, Llama-3.2-3B hit that on every round where it
called tools in parallel.

3. What happens when a tool-round limit is hit

Since #262, every provider stops a response after 8 tool rounds, or when a round repeats the previous one, by throwing
GenerationError.decodingFailure. What we observed:

  • The tool work is discarded. After a limit error the transcript holds only the prompt (88 of 88 errors on 1.0.0).
    .preserveTranscript keeps the tool entries only for streamResponse.
  • The delegate never sees the round that hits the limit, in 373 of 373 cases. So it can't .stop or substitute
    output there.
  • The error is hard to identify. Foundation Models documents decodingFailure as a deserialization failure, so
    callers can only recognize this case by matching on debugDescription.
  • The repeat rule mostly stops legitimate repeats.
    • We ran 336 responses with the assistant turn added (section 2), across 13 models. The rule stopped 118 of them.
    • 113 were legitimate: 74 re-checks of a job whose first check returned pending ("check again until it is
      done"), and 39 second die rolls when the prompt asked for two.
    • The loops that did run long (18 runs) were 5–8 different calls, which only the cap stops.
    • On plain 1.0.0, a one-argument polling tool is rejected on its first re-check every time (12/12).
  • Foundation Models allows these repeats. We ran the same tools and prompts through SystemLanguageModel, 30
    runs: no errors.
    • It re-checked the pending job with identical arguments and answered (12/12).
    • It rolled the die twice with identical calls (6/6).
    • Against a search tool that never matches, it stopped after 3–5 searches and said nothing was found (6/6).

This touches the question on #147 ("prevents llms to call exact same tool twice … Was it ever discussed?") and #280.

Questions

  1. Is the key-order dependence in the MLX repeat check (section 1) unintended? Should it compare calls the way
    ToolRoundLimit does?
  2. Is leaving the assistant tool-call turn (and tool_call_id/name) out of the MLX chat history (section 2)
    deliberate, for example for template compatibility? Or should the model see its own calls, as it does with
    llama.cpp and the remote providers?
  3. When a tool-round limit is hit, is throwing decodingFailure and losing the round's work the behavior you want?
    And should identical consecutive calls count as a loop at all, given that Foundation Models allows them?

Happy to share the harness, which logs every round through ToolExecutionDelegate, and the raw data.

Dominant language
Swift
Stars
948
Forks
99
Avg merge
19h 50m
Merged PRs (30d)
51

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/AnyLanguageModel

All issues in huggingface/AnyLanguageModel

Similar issues

More Swift issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.