MLXLanguageModel tool calling: flaky repeat check, missing assistant tool-call turn, and tool-round limits

Start in MLXLanguageModel.swift at the repeated-call check (around lines 1230 and 1496), where the key-order-dependent string is built, and compare it with ToolRoundLimit (#262). The issue also covers the missing assistant tool-call turn in pendingChat and the decodingFailure behaviour at the tool-round limit, which needs maintainer design decisions. Done would need the maintainers' answers to the three questions before any patch.

Version: 1.0.0 (80d50b0), mlx-swift-lm 3.32.3, MLX 4-bit models on an M3 Pro (macOS 27).

We measured how MLXLanguageModel handles multi-round tool calling with small local models, and found three things.
We'd like your view before going any further. Details are in the collapsed sections.

1. The repeated-call check depends on dictionary key order

MLXLanguageModel stops a response when a tool round repeats the previous round's calls. It compares strings built
with "\($0.function.name):\($0.function.arguments)", where arguments is a [String: MLXLMCommon.JSONValue]
(MLXLanguageModel.swift:1230-1233, :1496-1499).

A native Swift Dictionary seeds its hash table with its storage address (hashSeed(for:scale:) in the standard
library's HashTable.swift), so two identical calls can print their keys in different orders. As a result, the check
is reliable for tools with zero or one argument and a coin flip for tools with more.

previous: calculate:["a": MLXLMCommon.JSONValue.string("12"), "b": MLXLMCommon.JSONValue.string("7"), "operation": MLXLMCommon.JSONValue.string("multiply")]
current:  calculate:["b": MLXLMCommon.JSONValue.string("7"), "a": MLXLMCommon.JSONValue.string("12"), "operation": MLXLMCommon.JSONValue.string("multiply")]

import Foundation
let text = Data(#"{"pattern":"report","file":"notes.md","context":"0"}"#.utf8)
let rounds = try (0..<100).map { _ in try JSONDecoder().decode([String: String].self, from: text) }
print(Set(rounds.map { "\($0)" }).count)  // 6 (20 of 20 runs); 1 with SWIFT_DETERMINISTIC_HASHING=1 or a single key

2. The assistant turn that made a tool call is never sent back to the model

Between rounds, the chat is rebuilt from the transcript plus pendingChat, but the assistant turn carrying the calls
is never added:

• pendingChat gets .assistant(text) only when the round produced text (:1217-1219, :1485-1487).
• Results are appended as .tool(content), with no id and no name (:1267, :1525).
• convertTranscriptToMLXChat drops .toolCalls entries (:1737-1739).

So the chat template receives [system, user, tool]: a result with no call before it. What happens next depends on
the template:

• Gemma 4 (e2b, e4b, their QAT builds; the 26B-A4B QAT template behaves the same) drops the result.
  • The round-2 prompt is byte-identical to round 1, so the model repeats the same call until the repeat check or the
    8-round cap stops it.
  • Across 280 round transitions, the result never appeared in the prompt.
• gpt-oss-20b's template raises Message has tool role, but there was no previous assistant message with a tool call!. We checked this at the template level only.
• Ministral 3's template raises After the optional system message, conversation roles must alternate user and assistant roles except for tool calls and results. once two rounds in a response produced text (2 of 6 runs).
• Qwen3, Qwen3.5, Llama 3.2 and LFM2 render the result, but without the call that produced it.

3. What happens when a tool-round limit is hit

Since #262, every provider stops a response after 8 tool rounds, or when a round repeats the previous one, by throwing
GenerationError.decodingFailure. What we observed:

• The tool work is discarded. After a limit error the transcript holds only the prompt (88 of 88 errors on 1.0.0).
  .preserveTranscript keeps the tool entries only for streamResponse.
• The delegate never sees the round that hits the limit, in 373 of 373 cases. So it can't .stop or substitute
  output there.
• The error is hard to identify. Foundation Models documents decodingFailure as a deserialization failure, so
  callers can only recognize this case by matching on debugDescription.
• The repeat rule mostly stops legitimate repeats.
  • We ran 336 responses with the assistant turn added (section 2), across 13 models. The rule stopped 118 of them.
  • 113 were legitimate: 74 re-checks of a job whose first check returned pending ("check again until it is
    done"), and 39 second die rolls when the prompt asked for two.
  • The loops that did run long (18 runs) were 5–8 different calls, which only the cap stops.
  • On plain 1.0.0, a one-argument polling tool is rejected on its first re-check every time (12/12).
• Foundation Models allows these repeats. We ran the same tools and prompts through SystemLanguageModel, 30
  runs: no errors.
  • It re-checked the pending job with identical arguments and answered (12/12).
  • It rolled the die twice with identical calls (6/6).
  • Against a search tool that never matches, it stopped after 3–5 searches and said nothing was found (6/6).

This touches the question on #147 ("prevents llms to call exact same tool twice … Was it ever discussed?") and #280.

Questions

1. Is the key-order dependence in the MLX repeat check (section 1) unintended? Should it compare calls the way
   ToolRoundLimit does?
2. Is leaving the assistant tool-call turn (and tool_call_id/name) out of the MLX chat history (section 2)
   deliberate, for example for template compatibility? Or should the model see its own calls, as it does with
   llama.cpp and the remote providers?
3. When a tool-round limit is hit, is throwing decodingFailure and losing the round's work the behavior you want?
   And should identical consecutive calls count as a loop at all, given that Foundation Models allows them?

Happy to share the harness, which logs every round through ToolExecutionDelegate, and the raw data.