StirrupAgentRunner's default model id never matches a price-table key, so est_cost silently returns None

Open
#522 2 comments 0 reactions 1 assignee View on GitHub

@AmirF194 is already working on this.

Since Aug 21, 2026.

Assessment

This issue has not been assessed yet.

Description

Background

_estimate_cost (src/evaluation/metrics.py) looks up a model's per-token
rate in _PRICE_PER_1M after normalizing the model id through
_normalize_model. For StirrupAgentRunner's own default model,
watsonx/meta-llama/llama-4-maverick-17b-128e-instruct-fp8
(src/agent/stirrup_agent/runner.py), the normalized id never matches the
existing llama-4-maverick table entry, so _estimate_cost returns None
and any report built on it (including aggregate_ops's
est_input_cost_usd_total/est_output_cost_usd_total, added in #518) is
silently missing a dollar figure for runs using the default model.

src/evaluation/tests/test_metrics.py only exercises single-segment ids
(granite-13b, one date-suffix case), so this gap isn't caught by the
existing suite.

Steps to Reproduce
import sys
sys.path.insert(0, "src")
from evaluation import metrics

default_model = "watsonx/meta-llama/llama-4-maverick-17b-128e-instruct-fp8"
print("normalized:", metrics._normalize_model(default_model))
print("cost:", metrics._estimate_cost(default_model, 1000, 1000))
print("control (bare table key):", metrics._estimate_cost("llama-4-maverick", 1000, 1000))
Observed
normalized: llama-4-maverick-17b-128e-instruct-fp8
cost: None
control (bare table key): 0.00112
Expected

llama-4-maverick-17b-128e-instruct-fp8 resolves to the existing
llama-4-maverick entry, same as the control case.

Environment
  • Repo HEAD: 3a7bd23c7421f6ec3c0f215f018d7a8c2bcd8e37
  • Python 3.12.14, python:3.12-slim Docker image, pydantic from PyPI

Reproduced end to end in a clean container against current main, running
the real module (not a mock). Happy to send a PR if useful.

Dominant language
Python
Stars
2.3k
Forks
325
Avg merge
16h 33m
Merged PRs (30d)
14

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from IBM/AssetOpsBench

All issues in IBM/AssetOpsBench

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.