Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

count_tokens silently substitutes the GPT-4 tokenizer for every non-bare-OpenAI model name; error reaches +108% on CJK and feeds prompt trimming

Aperta
  • Qualcosa non va?
#340 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai, backend

Direzione di ricerca

Start with src/agentlab/llm/llm_utils.py and inspect get_tokenizer and count_tokens; then trace agents/dynamic_prompting.py::fit_tokens and its logging call. Run the existing token-count and dynamic-prompting tests to establish current behavior. Done means the fallback is visible or correctable as scoped in the issue, without changing existing default behavior.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

count_tokens(text, model=...) ignores model for any model string AgentLab's own litellm naming
produces, and silently substitutes GPT-4's tokenizer. I measured what that costs: 0% on code, +11.7%
on English prose, +18.5% on Japanese, +108.3% on Chinese, against a real non-OpenAI tokenizer.

This is deliberate — #254 made it so, to stop runs crashing on anthropic/claude-3.5-sonnet. I'm not
asking to revert that. I'm asking whether the workaround can be made visible and correctable, and
reporting the measurement, which I don't think has been done.

The resolution rate

@cache
def get_tokenizer(model_name="gpt-4"):
    try:
        return tiktoken.encoding_for_model(model_name)
    except KeyError:
        logging.info(f"Could not find a tokenizer for model {model_name}. Trying HuggingFace.")
    try:
        from transformers import AutoTokenizer
        return AutoTokenizer.from_pretrained(model_name)
    except Exception as e:
        logging.info(f"Could not find a tokenizer for model {model_name}: {e} Defaulting to gpt-4.")
    return tiktoken.encoding_for_model("gpt-4")

Executed against the current file:

model string tokenizer used
gpt-4 cl100k_base (correct)
gpt-4o o200k_base (correct)
openai/gpt-4o cl100k_base — silent fallback
gemini/gemini-2.5-pro cl100k_base — silent fallback
anthropic/claude-sonnet-4-20250514 cl100k_base — silent fallback
vertex_ai/gemini-2.5-pro cl100k_base — silent fallback

Six of seven fall back, including openai/gpt-4o — wrong even for OpenAI's own model, because
AgentLab uses provider/model naming that tiktoken does not recognise. The log line is INFO.

Measured cost

Versus Qwen/Qwen3-8B (a proxy — Gemini's tokenizer isn't publicly loadable, and per-vendor
magnitudes will differ):

content cl100k (what AgentLab reports) Qwen3 real error
Python code 480 480 0.0%
mixed + emoji 580 580 0.0%
English prose (agent logs) 451 511 +11.7%
Japanese 768 648 +18.5%
Chinese 1000 480 +108.3%

Reproduce:

import ast, logging, tiktoken
src = open("src/agentlab/llm/llm_utils.py").read()
fn = next(n for n in ast.parse(src).body if isinstance(n, ast.FunctionDef) and n.name == "get_tokenizer")
fn.decorator_list = []                      # drop @cache so each call re-resolves
ns = {"tiktoken": tiktoken, "logging": logging}
exec(compile(ast.Module(body=[fn], type_ignores=[]), "<f>", "exec"), ns)
ns["get_tokenizer"]("gemini/gemini-2.5-pro").name     # -> cl100k_base

Why it matters beyond tidiness

agents/dynamic_prompting.py::fit_tokens shrinks a prompt until the count fits:

n_token = count_tokens(prompt_str, model=model_name)
if n_token <= max_prompt_tokens:
    return prompt
shrinkable.shrink()

Because the fallback over-counts prose and CJK, the loop stops when the wrong count fits — so the
agent silently gets a smaller prompt than intended, by a content-dependent margin.

Small related inconsistency: the log line at dynamic_prompting.py:276 calls
count_tokens(prompt_str) without model=, so it reports a different tokenizer than the loop just used.

litellm.encode is not a drop-in fix

I expected litellm.encode(model=..., text=...) (litellm >=1.75.3 is already a dependency) to fix
this. It does not — on 1.104.0 it returns identical counts for all six model strings, and
unchanged after litellm.disable_hf_tokenizer_download = False. So I am not proposing it.

Suggestions, least intrusive first

  1. Log the substitution at WARNING instead of INFO.
  2. Record (model → tokenizer actually used) in run metadata, so affected runs can be excluded
    downstream.
  3. Add count_tokens(..., strict=True) that raises instead of substituting.

None of these change existing behaviour.

Scope of my claim

Not claiming this invalidates any published result. For English and code the error is ≤11.7% and
usually ~0%; and per the power floor in arXiv 2602.07150, a 2% success difference already needs ~9
runs/arm to detect. The CJK figure is large, but I have no evidence that any published browser-agent
result rests on CJK prompt trimming. Happy to be told this is a non-issue.

Lingua principale
Python
Stelle
644
Fork
134
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ServiceNow/AgentLab

Tutte le issue di ServiceNow/AgentLab

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.