AttributeError: input_cost_per_token when using an unregistered model (e.g. Fireworks DeepSeek V4 Flash)

Aperta
  • Qualcosa non va?
#86 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
72/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
api, backend

Direzione di ricerca

Inizia da get_model_info e Completion.cost in fastllm/types.py, quindi esamina cost() in openai_chat.py, openai_responses.py, anthropic.py e gemini.py. Esegui la riproduzione minima dell'issue e aggiungi o aggiorna la coverage per un modello non registrato. Il lavoro è completato quando i metadati di costo mancanti o parziali non causano più un'eccezione durante il monitoraggio dei costi dopo il completamento di una risposta.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Bug report: AttributeError: input_cost_per_token when using an unregistered model (e.g. Fireworks DeepSeek V4 Flash)

Repo: AnswerDotAI/fastllm
Type: Bug
Severity: Medium — crashes at the end of every turn for any model missing cost metadata, after the response has already been delivered.

Summary

Calling acomplete / AsyncChat with a valid vendor-prefixed model that is not present in either model_info_registry (fastllm/types.py) or the bundled litellm price DB (fastllm/model_prices.json) succeeds over the wire, streams the full response, and then crashes while tracking usage with:

AttributeError: input_cost_per_token

The metadata lookup returns an empty AttrDict (via dict2obj({})), and the API-specific cost() functions access pricing fields by attribute, e.g. openai_chat.py:

cost  = in_txt  * m.input_cost_per_token  + out_txt * m.output_cost_per_token

AttrDict raises AttributeError on missing attributes (.get() works fine, attribute access does not).

Concrete example

Model: accounts/fireworks/models/deepseek-v4-flash-0731 via fireworks_ai/ prefix — a valid, working model on the Fireworks API (the unversioned deepseek-v4-flash ID returns 404). It is absent from both metadata sources, so the crash fires on completion.

Verified with fastllm 0.0.41 + shell_sage 1.1.1; same code exists on main (checked 2026-08-17).

Minimal reproduction (no API key required)

from fastllm.types import get_model_info, Usage
from fastllm.openai_chat import cost

meta = get_model_info('accounts/fireworks/models/deepseek-v4-flash-0731', 'fireworks_ai')
assert not dict(meta)                     # metadata lookup misses

cost(Usage(prompt_tokens=10, completion_tokens=5,
           raw={'prompt_tokens': 10, 'completion_tokens': 5}), meta)
# AttributeError: input_cost_per_token

End-to-end path (requires an API key): AsyncChat._call → self._track(res) → UsageStats.from_response(res) → cost=r.cost → Completion.cost property (types.py) → api.cost(self.usage, meta).

Root cause

  • get_model_info(mn, vendor_name) (types.py) falls back to get_model_meta, which returns dict2obj({}) when the model is in neither model_info_registry nor litellm's model_prices.json.
  • cost() in all four API modules then does attribute access on that empty object:
    • openai_chat.py:202 — m.input_cost_per_token etc.
    • openai_responses.py:251 — via tier_rate(m, 'input_cost_per_token', tier)
    • anthropic.py:336 — m.input_cost_per_token
    • gemini.py:274 — via tier_rate(m, ...) (this one is partially guarded since tier_rate uses meta.get(...), but approx_pricing in types.py does p['input_cost_per_token'] → KeyError on empty dict)
  • The codebase is already defensive elsewhere against missing metadata — e.g. _prep_call uses model_info.get('max_output_tokens', 32_000) — but cost() is not.

Impact

  • Any model that is valid but unregistered crashes at the end of each turn, after the response has fully streamed — confusing for users (looks like the answer caused an error), and exit code becomes non-zero.
  • This includes common hosted models whose IDs are versioned (e.g. -0731) and thus never match the price DB.
  • max_output_tokens also silently degrades to the 32k fallback for these models.

Suggested fix

Make cost computation defensive when metadata is missing. Either:

  1. In types.py, Completion.cost property: if the resolved metadata dict is empty, return 0.0 (or skip cost entirely):
meta = get_model_info(self.model, self.vendor_name)
if not dict(meta): return 0.0
  1. And/or switch the per-API cost() implementations to .get(..., 0)/tier_rate() guards so a partial metadata dict can't crash either.

Workaround (until fixed)

Register the model via fastllm's public API at startup:

from fastllm.types import register_model_info, modern_llm
register_model_info(
    'accounts/fireworks/models/deepseek-v4-flash-0731', vendor_name='fireworks_ai', **modern_llm,
    max_input_tokens=1048576, max_output_tokens=384000, max_tokens=384000,
    input_cost_per_token=1.4e-07, output_cost_per_token=2.8e-07,
    cache_read_input_token_cost=2.8e-08,
)

(pricing values taken from the bundled litellm entry for fireworks_ai/accounts/fireworks/models/deepseek-v4-flash)

Lingua principale
Jupyter Notebook
Stelle
18
Fork
6
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di AnswerDotAI/fastllm

Tutte le issue di AnswerDotAI/fastllm

Issue simili

Altre issue su Backend & API Design

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.