avg_cost_usd / avg_credits divide by metered runs, not run count (avg × runs ≠ total)

Start in agent_cost_bench/models.py at BenchmarkRun.cost_stats_by_target(), especially the lines calculating avg_cost and avg_credits. Compare those denominators with n and the emitted runs value, then verify that the chosen fix makes the averages reconcile with totals and makes partial metering visible if the added counts are retained.

Summary

BenchmarkRun.cost_stats_by_target() computes avg_cost_usd and avg_credits by dividing by the number of runs that carry a cost/credit value, rather than by the run count n. If any run in a target's set reports no usage data, the reported per-run average is inflated and avg_cost_usd × runs no longer reconciles with total_cost_usd.

The JSON key is avg_cost_usd and the console column is Avg Cost/Run — both read as a mean over all runs.

Where

agent_cost_bench/models.py, on main @ 72f7586:

• L1055 — avg_cost = total_cost / len(costed) if costed else 0.0
• L1075 — avg_credits = total_credits / len(credit_vals) if credit_vals else 0.0

n = len(results) is already in scope. It is used for the neighbouring avg_latency = sum(latencies) / n and emitted as "runs": float(n) in the same dict — which is what suggests an oversight rather than a deliberate "mean over metered runs".

Observed

A model-compare run over 10 tasks per target, where exactly one run per target recorded cost_usd: null / total_credits: 0:

summary.total_cost_usd = 3.3408 is correct, but Σ(reported avg × 10) = 3.71 — about 11% high.

A cli-compare run of 9 tasks in the same session is unaffected, because there len(costed) == n. That coincidence is what isolates the denominator as the cause rather than the totals.

Impact

The per-run figure is the headline cost number in the console table and the HTML report, and it silently fails to reconcile with the run total. Nothing in the output signals that some runs were unmetered, so there is no indication the denominator differs from runs.

Suggested fix

Either:

1. Divide by n, so the average reconciles with the total (unmetered runs counting as zero), and expose the metered-run counts so partial metering is visible rather than silent; or
2. Keep the current denominator but rename the field (e.g. avg_cost_usd_per_metered_run) and surface costed_runs in the table and report.

I have a patch for option 1 — the two one-line changes plus costed_runs / credited_runs added to the stats dict. Inlining it here so it can be applied directly; happy to raise it as a PR instead if you'd prefer.

diff --git a/agent_cost_bench/models.py b/agent_cost_bench/models.py
--- a/agent_cost_bench/models.py
+++ b/agent_cost_bench/models.py
@@ -1052,7 +1052,7 @@ class BenchmarkRun:
             costed = [r for r in results if r.cost_usd is not None]
             total_cost = sum(r.cost_usd for r in costed)
             passed_cost = sum(r.cost_usd for r in passed if r.cost_usd is not None)
-            avg_cost = total_cost / len(costed) if costed else 0.0
+            avg_cost = total_cost / n if n else 0.0
             latencies = [
                 r.cli_reported_seconds if r.cli_reported_seconds else r.duration_seconds
                 for r in results
@@ -1072,7 +1072,7 @@ class BenchmarkRun:
             passed_credits = sum(
                 (r.native_credits or 0.0) for r in passed if r.native_credits is not None
             )
-            avg_credits = total_credits / len(credit_vals) if credit_vals else 0.0
+            avg_credits = total_credits / n if n else 0.0
 
             out[target] = {
                 "runs": float(n),
@@ -1089,6 +1089,8 @@ class BenchmarkRun:
                 "credits_per_pass": (passed_credits / len(passed)) if passed else float("inf"),
                 "has_credits": 1.0 if credit_vals else 0.0,
                 "has_cost": 1.0 if costed else 0.0,
+                "costed_runs": float(len(costed)),
+                "credited_runs": float(len(credit_vals)),
             }
         return out

Secondary, lower priority

cost_per_pass is passed_cost / len(passed), i.e. the mean cost of the passing runs. The column label Cost/Success reads more naturally as total spend ÷ successful tasks, which is a different number whenever failures cost differently than passes. Worth documenting the definition in the README or renaming the column, since the two are easy to conflate when quoting results.