Hosts that load tools lazily: what they need to know, and what pages should not do to avoid the cost

Read the related discussions in #133, #255, and #262, then check the README guidance and the maze demo linked in the issue. Work out whether the spec should add a deferLoading hint or use an existing shape, and how hosts should handle it. Done means the spec clearly defines the hint and its effect on tool loading.

Putting every tool definition in every prompt is expensive, and it's worst for local inference, which is what we work on. A laptop runs a small, often quantized model with a small context, and recomputing a long prompt on a laptop GPU takes real time. Hosts load tools lazily to avoid that. Anthropic's tool search does it for its API, Claude Code works the same way, and we use deferLoading heavily in our own local inference framework. I don't think many people here know how it works, so here goes.

The system prompt and the tool list never change. Deferred tools aren't in them. The model gets one extra tool, a search tool, and when it needs something it calls it. The tool result is the answer: the matching tools, schema included. It sits in the conversation like any other tool output, and the model can call those tools from then on. Nothing at the start of the prompt moved, and that's the whole point, because serving stacks only reuse cached work (the KV cache) for the longest identical start of a prompt. Change the start and you pay for everything after it, same as in #312.

That initial prefill can be very slow on local inference. So breaking the KV cache (the prefix cache) isn't a token cost like it is on cloud inference, it's a very visible slowdown for the user.

What the host sends, and it never changes:

  system prompt:  You are a shopping assistant.
                  Tools you can search for: cart.add_item, cart.apply_promo,
                  cart.checkout, account.login, account.orders
                  (one line each in the real thing)
  tools:          [ search_tools ]      <- the only tool that is always loaded
                                           the deferred tools are NOT here

What happens in the conversation, which only grows:

  user:       add the blue shirt to my cart
  model:      calls search_tools({ query: "cart.*" })
  result:     cart.add_item    { description: "Add a product to the cart",
                                 inputSchema: { productId: string, quantity: number } }
              cart.apply_promo { description: "...", inputSchema: { ... } }
              cart.checkout    { description: "...", inputSchema: { ... } }
              (the full definitions arrive here, as the tool result)
  model:      calls cart.add_item({ productId: "blue-shirt", quantity: 1 })
  result:     "Added 1 item. Cart total: $24."

This is a sketch, not a wire format. The list of names in the system prompt is written by the host. In Claude's API the search result is a list of tool_reference blocks that get expanded into the full definitions, which comes down to the same thing as above.

You can also look tools up by exact name, and a host can write its own search, which we have to, since local runtimes don't have the hosted one.

Local inference is where this bites hardest, and I think it's mostly missing from the discussions here. There's no hosted tool search, so the host builds it itself. The model is small, so it picks worse from a long list, and the context is small too. And, as above, a cache that keeps getting invalidated hurts. A page that works fine against a big cloud model can be unusable on a device.

The part nobody talks about is how the model knows what to search for. We put a short list in the system prompt: the exact name plus one line for each of our own tools. It works well. Page tools aren't in it, and can't be, because the page wrote that text and the system prompt is where the model trusts most. So for page tools the model guesses keywords, and that's worse. What I'd want is a short, length-limited line per tool that I can show as untrusted text. title might do it (#133), or a new field. Either way the spec should say not to put it in the system prompt.

Now the worry. Pages that list hundreds of tools make every prompt expensive. Pages that register and unregister tools to keep the list small change it all the time, and every change recomputes the rest of the prompt. Both are easy to do because registerTool() and its signal look free. The README even recommends the second one right now. The maze demo (cc @andreban) does a version of it, start_game first and then move, look and the rest, which is fine for a game with two states.

Adding a tool is easy for a lazy host, it joins the deferred set. Removing or changing one is harder, since the old definition is in the history and as far as I can tell the API rejects references to tools that are gone. A toolchange detail with added, removed and updated (#262) would help a lot.

So should pages be able to say a tool is fine to load on demand? Something like deferLoading, as a hint hosts can ignore. A page could list a lot without every host paying for all of it. #255 might already be the right shape.