[rig-decomposition-bench] Daily rig decomposition benchmark — 2026-10-30 — single-call

The prompt says the data is inline, but it contains no formulary rows, brand-to-generic lookup table, currency rates or 12 interaction and look-alike pairs. I have not invented any figures. There is no merged table, savings numbers or memo below. This is a data-driven pipeline that produces all three deliverables once the data is supplied.

**Needed from you:** the three hospital CSVs, the brand-to-generic lookup, FX rates, and the 12 pairs.

## Agent 1: normalization script (`formulary.py`)
```python
import csv, io, json, sys
from collections import defaultdict

FX = {'USD': 1.0}            # add rates from the provided data, e.g. 'EUR': x
UNIT_MG = {'mg': 1.0, 'g': 1000.0, 'mcg': 0.001}
BRAND2GENERIC = {}           # paste the provided lookup (lowercase brand -> generic)
PAIRS = []                   # paste the 12 pairs as ('drugA', 'drugB', 'type')
COLMAP = {                   # one entry per hospital: source column -> standard column
  'H1': {'drug': 'drug', 'strength': 'strength', 'unit': 'unit', 'form': 'form',
         'cost': 'cost', 'currency': 'currency', 'cost_basis': 'cost_basis',
         'units_per_pack': 'units_per_pack', 'class': 'class'},
  'H2': {}, 'H3': {}}
RAW = {'H1': '', 'H2': '', 'H3': ''}   # paste inline CSV text per hospital

def load():
    rows = []
    for h, text in RAW.items():
        for r in csv.DictReader(io.StringIO(text.strip())):
            m = {std: r[src].strip() for std, src in COLMAP[h].items() if src in r}
            m['hospital'] = h
            rows.append(m)
    return rows

def normalize(r):
    name = r['drug'].lower()
    r['generic'] = BRAND2GENERIC.get(name, name)
    r['strength_mg'] = float(r['strength']) * UNIT_MG[r['unit'].lower()]
    cost = float(r['cost']) * FX[r['currency'].upper()]
    # per-vial pricing is the base; per-mL pricing is converted with units_per_pack (mL per vial)
    if r.get('cost_basis', 'per_unit') == 'per_ml':
        cost *= float(r['units_per_pack'])
    r['cost_usd'] = round(cost, 4)
    r['form'] = r['form'].lower()
    return r

def main():
    rows = [normalize(r) for r in load()]
    groups = defaultdict(list)
    for r in rows:
        groups[(r['generic'], r['strength_mg'], r['form'])].append(r)
    merged, dup_log = [], []
    for k, g in groups.items():
        g.sort(key=lambda x: x['cost_usd'])
        keep = dict(g[0])
        keep['source_hospitals'] = sorted({x['hospital'] for x in g})
        merged.append(keep)
        for d in g[1:]:
            dup_log.append({'key': k, 'removed_from': d['hospital'],
                            'kept_from': g[0]['hospital'], 'all_sources': keep['source_hospitals']})
    assert all(r['cost_usd'] >= 0 for r in merged)
    assert len({(r['generic'], r['strength_mg'], r['form']) for r in merged}) == len(merged)

    # Agent 2: savings if every hospital buys the lowest-cost equivalent
    by_key = defaultdict(list)
    for r in rows:
        by_key[(r['generic'], r['strength_mg'], r['form'])].append(r)
    cls, opps = defaultdict(float), []
    for k, g in by_key.items():
        low = min(x['cost_usd'] for x in g)
        s = sum(x['cost_usd'] - low for x in g)
        cls[g[0]['class']] += s
        opps.append({'product': k, 'class': g[0]['class'], 'savings_usd': round(s, 2)})
    top5 = sorted(opps, key=lambda o: -o['savings_usd'])[:5]

    # Agent 3: flag only pairs where both drugs are present
    present = {r['generic'] for r in merged}
    flags = [p for p in PAIRS if p[0] in present and p[1] in present]

    json.dump({'merged': merged, 'duplicates': dup_log,
               'class_savings': {c: round(v, 2) for c, v in cls.items()},
               'top5': top5, 'safety_flags': flags},
              sys.stdout, indent=2, default=str)

main()
```

## Agent 4: memo template (about 300 words, filled from the JSON output)
**Memo: Unified Formulary Consolidation**
- **Summary:** N unique products after removing D duplicates. All costs are in USD and all strengths in mg.
- **Savings:** Adopting the lowest-cost equivalent saves $X in total. Per-class totals are in `class_savings`. The top 5 opportunities are listed from `top5`.
- **Safety:** F of the 12 pairs have both drugs present and need pharmacy and therapeutics review: [list]. Pairs with a missing drug are not flagged.
- **Recommendations:** (1) Adopt the lowest-cost products for the top 5 opportunities. (2) Apply look-alike/sound-alike controls such as tall-man lettering and separate storage to the flagged pairs. (3) Review interaction alerts in the electronic health record for the flagged pairs. (4) Phase the rollout by class.
- **Data-quality caveats:** (1) FX rates are a static snapshot. (2) Per-mL to per-vial conversion depends on the stated pack size. (3) Brand-to-generic mapping covers only the provided lookup, so unmapped brands remain separate. (4) Equivalence is judged by name, strength and form only, with no bioequivalence check. (5) Savings assume equal usage volumes, which the data does not give.

## Deliverable 1: Script and duplicate log
The task input did not include the data needed to produce the script, merged table, or duplicate log. The three hospital CSV blocks, the brand-to-generic lookup, the currency rates and the 12 drug-pair list were all missing. I did not invent any, because the merged table and duplicate log would be fabricated. I have logged the missing data. Please resend the inline data and I will produce the script, then '=== MERGED TABLE ===' and '=== DUPLICATE LOG ==='.

## Deliverable 2: Merged table


## Savings analysis


## Safety review


## Deliverable 3: Executive memo
EXECUTIVE MEMO

SUMMARY
This memo was meant to combine the savings analysis and the safety review into one page. Both inputs arrived empty. The savings analysis section and the safety review section contain no content. I cannot responsibly report savings or safety findings. Any figures I gave would be invented, which the brief prohibits.

KEY SAVINGS FIGURES
None available. No total savings, no top 5 savings opportunities and no per-item amounts were provided. Once the savings analysis is supplied, I will report its numbers exactly as given, including the top 5.

SAFETY FLAGS
None available. The safety review contained no findings, ratings or flagged items. I cannot confirm that any savings opportunity is free of patient-safety risk. Until the review is received, treat every savings initiative as not yet cleared for safety.

RECOMMENDATIONS
1. Resend both inputs: the full savings analysis (total and top 5 opportunities) and the complete safety review.
2. Hold all decisions on cost-reduction initiatives until the savings and safety findings can be read side by side.
3. Once the inputs arrive, check each top-5 savings item against the safety flags. Prioritize items with no safety concerns, and send any flagged item for clinical review before approval.
4. Have the analysis owner and the safety reviewer confirm their outputs before the memo is circulated.

DATA-QUALITY CAVEATS
1. Missing source data: both the savings analysis and the safety review were empty, so no figure or finding in this memo is verified.
2. No methodology: without the analysis, I cannot say how savings were calculated, over what period, or against what baseline.
3. No linkage between sources: it is unknown whether the savings items and safety flags use the same identifiers or categories. This could prevent reliable cross-referencing.
4. Possible transmission error: the empty inputs may reflect a formatting or export failure rather than a true absence of findings. An empty safety review should not be read as "no safety concerns."

NEXT STEP
On receipt of the complete inputs, a final one-page memo will be issued within the 400-word limit.

import { agent, workflow, s, repair, configureAgent, copilotEngine } from "rig";

configureAgent(copilotEngine({ server: true }));

const DEADLINE = Date.now() + 25 * 60_000;
const remaining = () => DEADLINE - Date.now();

const SKELETON = `import { agent, workflow, s } from "rig";

// Agent role: <role>
const analyze = agent({ model: "small", instructions: "...", output: s.string });

// Workflow role: combines the agents' outputs into the final solution
export default workflow({
  meta: { name: "decomposed-solution", description: "..." },
  body: async ({ call }) => {
    const analysis = await call(analyze, "...");
    return { solution: analysis ?? "" };
  },
});`;

const CONTRACT = `Write a complete, self-contained rig TypeScript program. Rules:
- Import only from "rig" (agent, workflow, s).
- Use at least two agents, each preceded by a "// Agent role: ..." comment; use model "small" for simple sub-steps and "medium" or "large" for harder ones. The final answer must combine their outputs.
- The root must take no required input and be exported with "export default" without being invoked.
- A workflow() spec accepts only meta, an optional input, and body. It has no output or execute field; the final { solution } object is simply what body returns. An agent() root instead declares output: s.object({ solution: s.string }).
- Use output: s.string for every agent producing source text or free-form solution text; never wrap large text in an object schema.
- call(agent, "prompt string") returns string or null.
- Do not use console.log. Return ONLY the TypeScript source, no Markdown fences.
Required shape (abridged to one agent; yours needs at least two):
${SKELETON}`;

// Agent role: pick a benchmark task
const picker = agent({
  model: "small",
  maxTurns: 2,
  addons: repair(),
  instructions: `Pick one concrete, complicated task a person or team might face in a single day that would naturally benefit from being split across sub-agents running different models: multi-source research and synthesis, a multi-file coding task with design/implementation/review phases, a structured report combining independent analyses, or a multi-step data transformation pipeline. It must be self-contained (no external files or live web access) and concrete enough to grade. Vary the domain randomly (today is ${new Date().toISOString()}). Return title, domain, description, and 3-6 concrete checkable success criteria.`,
  output: s.object({
    title: s.string,
    domain: s.string,
    description: s.string,
    criteria: s.array(s.string),
  }),
});

// Agent role: solve the whole task alone in one call
const solver = agent({
  model: "medium",
  maxTurns: 1,
  instructions: "Solve the entire task yourself in one response, addressing every success criterion. No delegation, no tools. Return the full solution (Markdown and code fences allowed).",
  output: s.string,
});

// Agent role: write the decomposed rig program
const writer = agent({
  model: "medium",
  maxTurns: 1,
  instructions: CONTRACT,
  output: s.string,
});

// Agent role: fix or regenerate the decomposed rig program from an error
const fixer = agent({
  model: "medium",
  maxTurns: 1,
  instructions: CONTRACT + "\nYou fix rig programs given the exact error output, changing precisely what the error names and preserving what already worked. If told there is no source, generate the full program from scratch.",
  output: s.string,
});

// Agent role: grade both solutions
const grader = agent({
  model: "large",
  maxTurns: 1,
  addons: repair(),
  instructions: "Score each solution 0-10 on how completely and correctly it satisfies the success criteria. Judge only correctness and completeness of content, not length and not which approach produced it. Pick winner: single-call, decomposed, or tie, and explain the comparison.",
  output: s.object({
    singleScore: s.number,
    decomposedScore: s.number,
    winner: s.enum("single-call", "decomposed", "tie"),
    rationale: s.string,
  }),
});

type Attempt = { attempt: number; generation: string; typecheck: string; execute: string; error: string; note: string };

async function runCli(args: string[], source: string, timeoutMs: number): Promise<{ ok: boolean; stdout: string; output: string }> {
  const { spawn } = await import("node:child_process");
  return new Promise((resolve) => {
    const child = spawn("node", ["rig.ts", ...args], { cwd: process.cwd() });
    let out = "";
    let err = "";
    let timedOut = false;
    const timer = setTimeout(() => { timedOut = true; child.kill("SIGKILL"); }, Math.max(1000, timeoutMs));
    child.stdout.on("data", (d: Buffer) => { out += d.toString(); });
    child.stderr.on("data", (d: Buffer) => { err += d.toString(); });
    child.on("error", (e: Error) => { clearTimeout(timer); resolve({ ok: false, stdout: out, output: `spawn error: ${e.message}` }); });
    child.on("close", (code: number | null) => {
      clearTimeout(timer);
      const output = (err + (code === 0 ? "" : out)).trim() + (timedOut ? "\n[timed out]" : "");
      resolve({ ok: code === 0 && !timedOut, stdout: out, output });
    });
    child.stdin.end(source);
  });
}

function stripFences(src: string): string {
  const m = src.match(/```(?:ts|typescript)?\n([\s\S]*?)```/);
  return (m ? m[1] : src).trim();
}

// Workflow role: run the benchmark end to end and return the report
export default workflow({
  meta: { name: "decomposition-benchmark", description: "Compare single-call vs decomposed solutions" },
  body: async ({ call }) => {
    const task = await call(picker, "Pick the task now.");
    if (!task) return { status: "fail", error: "task picker returned null" };
    const taskText = `Title: ${task.title}\nDomain: ${task.domain}\nDescription: ${task.description}\nSuccess criteria:\n${task.criteria.map((c: string) => `- ${c}`).join("\n")}`;

    const t0 = Date.now();
    const single = (await call(solver, taskText)) ?? "";
    const singleMs = Date.now() - t0;

    const t1 = Date.now();
    const attempts: Attempt[] = [];
    let source = "";
    let decomposed = "";
    let status = "fail";
    let prev = { source: "", error: "" };
    for (let n = 1; n <= 2; n++) {
      if (remaining() < 3 * 60_000) break;
      const rec: Attempt = { attempt: n, generation: "", typecheck: "skipped", execute: "skipped", error: "", note: "" };
      let gen: string | null;
      if (n === 1) {
        gen = await call(writer, `Task:\n${taskText}\n\nWrite the rig program that solves this task by decomposition.`);
      } else if (prev.source.trim()) {
        gen = await call(fixer, `Task:\n${taskText}\n\nCurrent source:\n${prev.source}\n\nExact captured error:\n${prev.error}\n\nFix precisely what the error names, preserving what already worked.`);
      } else {
        gen = await call(fixer, `Task:\n${taskText}\n\nThere is no source to fix: the previous attempt produced nothing. Generate the complete program from the task and the skeleton.`);
      }
      const src = stripFences(gen ?? "");
      if (!src) {
        rec.generation = "empty";
        rec.error = "writer returned null/empty source";
        rec.note = n === 1 ? "second attempt regenerates from scratch" : "";
        attempts.push(rec);
        prev = { source: "", error: rec.error };
        continue;
      }
      rec.generation = "ok";
      source = src;
      const tc = await runCli(["--typecheck"], src, Math.min(2 * 60_000, remaining()));
      if (!tc.ok) {
        rec.typecheck = "fail";
        rec.error = tc.output;
        rec.note = n === 1 ? "next attempt fixes the typecheck error" : "";
        attempts.push(rec);
        prev = { source: src, error: tc.output };
        continue;
      }
      rec.typecheck = "pass";
      if (remaining() < 60_000) { rec.execute = "skipped (budget)"; attempts.push(rec); break; }
      const ex = await runCli(["--server"], src, Math.min(5 * 60_000, remaining()));
      if (!ex.ok) {
        rec.execute = "fail";
        rec.error = ex.output;
        rec.note = n === 1 ? "next attempt fixes the execution error" : "";
        attempts.push(rec);
        prev = { source: src, error: ex.output };
        continue;
      }
      rec.execute = "pass";
      try {
        const parsed = JSON.parse(ex.stdout);
        decomposed = typeof parsed?.solution === "string" ? parsed.solution : typeof parsed === "string" ? parsed : JSON.stringify(parsed);
      } catch {
        decomposed = ex.stdout.trim();
      }
      status = decomposed.trim() ? "pass" : "fail";
      attempts.push(rec);
      break;
    }
    const decomposedMs = Date.now() - t1;

    const graded = await call(grader, `${taskText}\n\n=== SOLUTION A (single-call) ===\n${single || "(empty)"}\n\n=== SOLUTION B (decomposed) ===\n${decomposed.trim() || "(decomposition produced no solution)"}`);

    return { status, task, singleMs, decomposedMs, single, decomposed, attempts, source, grading: graded };
  },
});