BM25SearchStrategy (the default) never matches underscore-joined terms in tool descriptions or param descriptions

Inizia da livekit-agents/livekit/agents/beta/toolsets/tool_search.py in tokenize (intorno alle righe 324-334), dove le chiavi di nome e parametro ricevono la sostituzione del trattino basso con spazio, ma i token di descrizione e di valore dei parametri no. Riproduci il problema con lo script BM25 vs KeywordSearchStrategy presente nell'issue, poi applica la stessa .replace("", " ") ai token di descrizione e di valore dei parametri. È completato quando le quattro query restituiscono get_records da BM25 e tests/test_tool_search.py continua a passare, più un nuovo test di regressione con un termine della descrizione unito da trattini bassi.

Bug Description

BM25SearchStrategy — the default search strategy for ToolSearchToolset (livekit-agents/livekit/agents/beta/toolsets/tool_search.py:79) — never matches underscore-joined terms that appear in a tool's description or in a parameter description (the values of the parameters dict).

search() normalizes the query with query.lower().replace("_", " ").split() (tool_search.py:307), and _tokenize applies that same underscore→space split to the tool name (line 326) and to parameter keys (line 330) — but not to description tokens (line 327) or parameter value tokens (line 331). So a description containing checking_balance is indexed as the single token checking_balance, while no query can ever produce that token (every query has _ split into spaces at line 307). Any search for checking_balance, checking balance, pending_refunds, pending refunds, … scores 0 and the tool is silently not returned.

The sibling KeywordSearchStrategy returns the same tool for the same query, and a control case where the description uses a space instead (checking balance) matches fine on BM25 — so this is specific to underscore-joined terms in descriptions/param descriptions.

This hits the LLM-facing path: with the default strategy, a tool_search call with query="checking_balance" raises ToolError: No tools found matching 'checking_balance'. even though the tool's description contains that exact term.

Expected Behavior

BM25SearchStrategy should return a tool whose description or parameter description contains the query terms, matching KeywordSearchStrategy's behaviour for the same input.

Basis:

• The BM25SearchStrategy docstring (tool_search.py:260-261): "Each SearchItem is treated as a document composed of its name (weighted 3x), description (weighted 2x), and parameter names/descriptions (weighted 1x)."
• search()'s own query normalization at tool_search.py:307 (query.lower().replace("_", " ")) shows the intended tokenization, and makes underscore-joined description tokens unreachable by any query.
• The LLM-facing query contract _DEFAULT_QUERY_DESCRIPTION (tool_search.py:51-53): "keywords to search for in the tool names and descriptions, split by spaces".

Reproduction Steps

No network or credentials needed — pure framework-internal search. With livekit-agents 1.8.3 (built from source at commit 1257fe0f987b53054e1d31b98e514991776f85d4), Python 3.12.14:

from livekit.agents.beta.toolsets.tool_search import (
    BM25SearchStrategy,
    KeywordSearchStrategy,
    SearchItem,
)


def run(label, items, query):
    kw_items = [SearchItem(source=i.source, name=i.name, description=i.description, parameters=i.parameters) for i in items]
    KeywordSearchStrategy().build_index(kw_items)
    kw_res = [i.name for i in KeywordSearchStrategy().search(query, kw_items, 5)]

    bm_items = [SearchItem(source=i.source, name=i.name, description=i.description, parameters=i.parameters) for i in items]
    BM25SearchStrategy().build_index(bm_items)
    bm_res = [i.name for i in BM25SearchStrategy().search(query, bm_items, 5)]
    print(f"[{label}] query={query!r:20} Keyword -> {kw_res}   BM25 -> {bm_res}")


# case 1: underscore-joined term only in the DESCRIPTION
desc_items = [
    SearchItem(
        name="get_records",
        description="Fetch the user's checking_balance from the ledger",
        parameters={"record_type": "Which records to fetch"},
        source="src_a",
    ),
    SearchItem(
        name="get_weather",
        description="Get current weather for a city",
        parameters={"city": "City name"},
        source="src_b",
    ),
]
run("desc-only", desc_items, "checking_balance")
run("desc-only", desc_items, "checking balance")

# case 2: underscore-joined term only in a parameter VALUE
val_items = [
    SearchItem(
        name="get_records",
        description="Fetch ledger records for the user",
        parameters={"filter": "One of: pending_refunds only, settled only, all"},
        source="src_a",
    ),
    SearchItem(
        name="get_weather",
        description="Get current weather for a city",
        parameters={"city": "City name"},
        source="src_b",
    ),
]
run("param-value", val_items, "pending_refunds")
run("param-value", val_items, "pending refunds")

Actual output:

[desc-only] query='checking_balance'   Keyword -> ['get_records']   BM25 -> []
[desc-only] query='checking balance'   Keyword -> ['get_records']   BM25 -> []
[param-value] query='pending_refunds'    Keyword -> ['get_records']   BM25 -> []
[param-value] query='pending refunds'    Keyword -> ['get_records']   BM25 -> []

Going through ToolSearchToolset with the default strategy (a @function_tool-decorated get_records whose docstring is """Fetch the user's checking_balance from the ledger."""), await ts._handle_search({"query": "checking_balance"}) — i.e. the tool_search function the LLM calls — raises:

livekit.agents.llm.tool_context.ToolError: No tools found matching 'checking_balance'.

at tool_search.py:153, while the same toolset built with search_strategy=KeywordSearchStrategy() loads the tool and returns "Tools loaded successfully.".

Operating System

macOS (darwin 25.6.0, arm64). No OS-specific code involved in the affected module.

Models Used

N/A — reproduces without any model; this is framework-internal search over SearchItems.

Package Versions

livekit-agents==1.8.3 (built from source at commit 1257fe0f987b53054e1d31b98e514991776f85d4; the affected file is unchanged on current main)
Python 3.12.14

Proposed Solution

In _tokenize (tool_search.py:324-334), apply the same .replace("_", " ") normalization to desc_tokens (line 327) and to the parameter VALUE tokens (line 331), matching the existing name/key handling and the query normalization in search():

desc_tokens = item.description.lower().replace("_", " ").split()
param_tokens.extend(v.lower().replace("_", " ").split())

I verified locally that with these two changes the repro above returns get_records from BM25 for all four queries, and the existing tests/test_tool_search.py suite (40 tests) still passes. Happy to open a PR with this change plus a regression test.

Additional Context

Root cause: tool_search.py:327 (desc_tokens = item.description.lower().split()) and tool_search.py:331 (param_tokens.extend(v.lower().split())) omit the underscore→space split that is applied to the name (tool_search.py:326), parameter keys (tool_search.py:330) and the query (tool_search.py:307), so description/param-description tokens containing _ are unreachable by any query.

Notes:

• ToolSearchToolset defaults to BM25SearchStrategy (tool_search.py:79), so every tool_search call the LLM makes goes through this path (_handle_search → _search_tools → strategy.search).
• Tool descriptions in Python routinely reference snake_case parameter/function names, so dynamic tool discovery silently misses those tools for those terms.
• The existing tests/test_tool_search.py only uses space-separated description terms, so this case is uncovered — all 40 tests pass at this commit.
• I searched the tracker for existing reports about tool_search / BM25 and found none.