Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Hosts that load tools lazily: what they need to know, and what pages should not do to avoid the cost

Aperta
  • Qualcosa non va?
#357 1 commento 1 reazione 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
javascript
Ambito
api, developer-experience

Direzione di ricerca

Read the related discussions in #133, #255, and #262, then check the README guidance and the maze demo linked in the issue. Work out whether the spec should add a deferLoading hint or use an existing shape, and how hosts should handle it. Done means the spec clearly defines the hint and its effect on tool loading.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Putting every tool definition in every prompt is expensive, and it's worst for local inference, which is what we work on. A laptop runs a small, often quantized model with a small context, and recomputing a long prompt on a laptop GPU takes real time. Hosts load tools lazily to avoid that. Anthropic's tool search does it for its API, Claude Code works the same way, and we use deferLoading heavily in our own local inference framework. I don't think many people here know how it works, so here goes.

The system prompt and the tool list never change. Deferred tools aren't in them. The model gets one extra tool, a search tool, and when it needs something it calls it. The tool result is the answer: the matching tools, schema included. It sits in the conversation like any other tool output, and the model can call those tools from then on. Nothing at the start of the prompt moved, and that's the whole point, because serving stacks only reuse cached work (the KV cache) for the longest identical start of a prompt. Change the start and you pay for everything after it, same as in #312.

That initial prefill can be very slow on local inference. So breaking the KV cache (the prefix cache) isn't a token cost like it is on cloud inference, it's a very visible slowdown for the user.

What the host sends, and it never changes:

  system prompt:  You are a shopping assistant.
                  Tools you can search for: cart.add_item, cart.apply_promo,
                  cart.checkout, account.login, account.orders
                  (one line each in the real thing)
  tools:          [ search_tools ]      <- the only tool that is always loaded
                                           the deferred tools are NOT here

What happens in the conversation, which only grows:

  user:       add the blue shirt to my cart
  model:      calls search_tools({ query: "cart.*" })
  result:     cart.add_item    { description: "Add a product to the cart",
                                 inputSchema: { productId: string, quantity: number } }
              cart.apply_promo { description: "...", inputSchema: { ... } }
              cart.checkout    { description: "...", inputSchema: { ... } }
              (the full definitions arrive here, as the tool result)
  model:      calls cart.add_item({ productId: "blue-shirt", quantity: 1 })
  result:     "Added 1 item. Cart total: $24."

This is a sketch, not a wire format. The list of names in the system prompt is written by the host. In Claude's API the search result is a list of tool_reference blocks that get expanded into the full definitions, which comes down to the same thing as above.

You can also look tools up by exact name, and a host can write its own search, which we have to, since local runtimes don't have the hosted one.

Local inference is where this bites hardest, and I think it's mostly missing from the discussions here. There's no hosted tool search, so the host builds it itself. The model is small, so it picks worse from a long list, and the context is small too. And, as above, a cache that keeps getting invalidated hurts. A page that works fine against a big cloud model can be unusable on a device.

The part nobody talks about is how the model knows what to search for. We put a short list in the system prompt: the exact name plus one line for each of our own tools. It works well. Page tools aren't in it, and can't be, because the page wrote that text and the system prompt is where the model trusts most. So for page tools the model guesses keywords, and that's worse. What I'd want is a short, length-limited line per tool that I can show as untrusted text. title might do it (#133), or a new field. Either way the spec should say not to put it in the system prompt.

Now the worry. Pages that list hundreds of tools make every prompt expensive. Pages that register and unregister tools to keep the list small change it all the time, and every change recomputes the rest of the prompt. Both are easy to do because registerTool() and its signal look free. The README even recommends the second one right now. The maze demo (cc @andreban) does a version of it, start_game first and then move, look and the rest, which is fine for a game with two states.

Adding a tool is easy for a lazy host, it joins the deferred set. Removing or changing one is harder, since the old definition is in the history and as far as I can tell the API rejects references to tools that are gone. A toolchange detail with added, removed and updated (#262) would help a lot.

So should pages be able to say a tool is fine to load on demand? Something like deferLoading, as a hint hosts can ignore. A page could list a lot without every host paying for all of it. #255 might already be the right shape.

Lingua principale
Bikeshed
Stelle
4.7k
Fork
328
Merge medio
2g 15h
PR unite (30g)
16

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di webmachinelearning/webmcp

Tutte le issue di webmachinelearning/webmcp

Issue simili

Altre issue su Backend & API Design

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.