Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Web search: Create tests and optimise tool use behaviour

Aperta
  • Qualcosa non va?
#694 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 5 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Ambito
ai-infra-agents, testing

Direzione di ricerca

The issue describes testing and optimizing an AI assistant's web search and fetch tool behavior. Start by examining the existing test suite for tool usage, likely in a tests/ directory. Look for prompts and configurations controlling max_content_tokens, max_uses, and URL allowlists. To assess the model's behavior, you'll need to run multi-turn conversation tests, observing the sequence of fetch and search calls and checking for redundant fetches. 'Done' means having a set of tests that surface the described issues and data to inform prompt or parameter adjustments.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

We need to create tests that surface how the assistant uses the new search/fetch tools and how it uses their outputs in its answers.

We need to test the assistant's behaviour when using the new tools to:

  • Assess if the tool is called at appropriate times, and not too often, as this could slow down model conversation turns. Prompts may need adjusting.
  • Optimise max_content_tokens (together with max_uses). It is currently quite low and might truncate pages before the relevant information too often. If we can optimise the assistant's behaviour to only use the tools when necessary, and not speculatively, we can tolerate higher token costs from fetched content.
  • Assess if the model knows how to navigate allowed/banned urls, and doesn't waste resources trying to open banned urls. Testing so far suggests it does respect the allowlist when the prompt names it.
  • Assess the separate problem of url_not_in_prior_context. Anthropic's web fetch will only open a URL that has already appeared in the conversation, which means a URL in a user message, a client side tool result, or an earlier search or fetch result. A URL in the system prompt does not count, and neither does one the model produced itself. The model does not appear to know this, so it opens every turn by trying to fetch a URL it remembers, being refused, and only then searching. The observed sequence is fetch → refused → fetch → refused → search → fetch success.
    • Two possible fixes, worth testing both: prompting it to search before fetching, or injecting relevant URLs into the user turn, which satisfies the rule directly. Note that removing the example URL from the prompt was already tested and did not fix this, because the model reaches for URLs it knows from training. Injecting a URL in the user turn was tested and did work on the first attempt.
  • Assess multi-turn conversations as well: is the assistant re-fetching the same content every turn? This may be slow and costly. A cheap patch for this could be to prompt it to state its findings in its answer to the user, the results would persist across conversation turns.
    • For background, return_history keeps only strings, so all fetched content is discarded at the end of every turn. The URL is discarded too, and a URL that appears only in Claude's own earlier output does not satisfy the prior context rule, so a follow-up question has to search and fetch again from scratch.
Lingua principale
Jupyter Notebook
Stelle
5
Fork
10
Merge medio
2g 12h
PR unite (30g)
17

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di OpenFn/apollo

Tutte le issue di OpenFn/apollo

Issue simili

Altre issue su AI Infra & Agents

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.