Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

count_tokens silently substitutes the GPT-4 tokenizer for every non-bare-OpenAI model name; error reaches +108% on CJK and feeds prompt trimming

Aberta
  • Algo errado?
#340 0 comentários 0 reações 0 responsáveis Ver no GitHub

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
48/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
python
Domínio
ai, backend

Direção de pesquisa

Start with src/agentlab/llm/llm_utils.py and inspect get_tokenizer and count_tokens; then trace agents/dynamic_prompting.py::fit_tokens and its logging call. Run the existing token-count and dynamic-prompting tests to establish current behavior. Done means the fallback is visible or correctable as scoped in the issue, without changing existing default behavior.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Summary

count_tokens(text, model=...) ignores model for any model string AgentLab's own litellm naming
produces, and silently substitutes GPT-4's tokenizer. I measured what that costs: 0% on code, +11.7%
on English prose, +18.5% on Japanese, +108.3% on Chinese, against a real non-OpenAI tokenizer.

This is deliberate — #254 made it so, to stop runs crashing on anthropic/claude-3.5-sonnet. I'm not
asking to revert that. I'm asking whether the workaround can be made visible and correctable, and
reporting the measurement, which I don't think has been done.

The resolution rate

@cache
def get_tokenizer(model_name="gpt-4"):
    try:
        return tiktoken.encoding_for_model(model_name)
    except KeyError:
        logging.info(f"Could not find a tokenizer for model {model_name}. Trying HuggingFace.")
    try:
        from transformers import AutoTokenizer
        return AutoTokenizer.from_pretrained(model_name)
    except Exception as e:
        logging.info(f"Could not find a tokenizer for model {model_name}: {e} Defaulting to gpt-4.")
    return tiktoken.encoding_for_model("gpt-4")

Executed against the current file:

model string tokenizer used
gpt-4 cl100k_base (correct)
gpt-4o o200k_base (correct)
openai/gpt-4o cl100k_base — silent fallback
gemini/gemini-2.5-pro cl100k_base — silent fallback
anthropic/claude-sonnet-4-20250514 cl100k_base — silent fallback
vertex_ai/gemini-2.5-pro cl100k_base — silent fallback

Six of seven fall back, including openai/gpt-4o — wrong even for OpenAI's own model, because
AgentLab uses provider/model naming that tiktoken does not recognise. The log line is INFO.

Measured cost

Versus Qwen/Qwen3-8B (a proxy — Gemini's tokenizer isn't publicly loadable, and per-vendor
magnitudes will differ):

content cl100k (what AgentLab reports) Qwen3 real error
Python code 480 480 0.0%
mixed + emoji 580 580 0.0%
English prose (agent logs) 451 511 +11.7%
Japanese 768 648 +18.5%
Chinese 1000 480 +108.3%

Reproduce:

import ast, logging, tiktoken
src = open("src/agentlab/llm/llm_utils.py").read()
fn = next(n for n in ast.parse(src).body if isinstance(n, ast.FunctionDef) and n.name == "get_tokenizer")
fn.decorator_list = []                      # drop @cache so each call re-resolves
ns = {"tiktoken": tiktoken, "logging": logging}
exec(compile(ast.Module(body=[fn], type_ignores=[]), "<f>", "exec"), ns)
ns["get_tokenizer"]("gemini/gemini-2.5-pro").name     # -> cl100k_base

Why it matters beyond tidiness

agents/dynamic_prompting.py::fit_tokens shrinks a prompt until the count fits:

n_token = count_tokens(prompt_str, model=model_name)
if n_token <= max_prompt_tokens:
    return prompt
shrinkable.shrink()

Because the fallback over-counts prose and CJK, the loop stops when the wrong count fits — so the
agent silently gets a smaller prompt than intended, by a content-dependent margin.

Small related inconsistency: the log line at dynamic_prompting.py:276 calls
count_tokens(prompt_str) without model=, so it reports a different tokenizer than the loop just used.

litellm.encode is not a drop-in fix

I expected litellm.encode(model=..., text=...) (litellm >=1.75.3 is already a dependency) to fix
this. It does not — on 1.104.0 it returns identical counts for all six model strings, and
unchanged after litellm.disable_hf_tokenizer_download = False. So I am not proposing it.

Suggestions, least intrusive first

  1. Log the substitution at WARNING instead of INFO.
  2. Record (model → tokenizer actually used) in run metadata, so affected runs can be excluded
    downstream.
  3. Add count_tokens(..., strict=True) that raises instead of substituting.

None of these change existing behaviour.

Scope of my claim

Not claiming this invalidates any published result. For English and code the error is ≤11.7% and
usually ~0%; and per the power floor in arXiv 2602.07150, a 2% success difference already needs ~9
runs/arm to detect. The CJK figure is large, but I have no evidence that any published browser-agent
result rests on CJK prompt trimming. Happy to be told this is a non-issue.

Linguagem predominante
Python
Estrelas
644
Forks
134
Nenhum PR com merge em 30d

Preparar o ambiente

Este projeto não oferece contêiner de desenvolvimento, Dockerfile nem guia de contribuição, então a configuração fica por sua conta: comece pelo README e veja nosso guia da primeira contribuição para os passos gerais.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de ServiceNow/AgentLab

Todas as issues de ServiceNow/AgentLab

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.