[rig-decomposition-bench] Daily rig decomposition benchmark — 2026-10-23 — single-call

The issue documents a benchmark workflow and its single-call and decomposed results; the benchmark source is described but not included in full. Start by reading the rig workflow in the issue and the linked Daily Rig Decomposition Benchmark run. Done means the benchmark's reported deliverables and verdict are reproducible from the complete source, which the issue notes is currently omitted or summarized.

Task

Title: Hospital Bed-Capacity Forecasting Pipeline with Independent Methodology Review
Domain: healthcare operations / data engineering

Description: A regional hospital network needs a one-day deliverable forecasting next-week ICU bed demand from a small synthetic dataset embedded in the task: 12 weeks of daily admissions for hospitals A, B and C. B has 4 missing days, C has a duplicated date row, and A has an admission count of -7. Phases: data cleaning; forecasting (7-day seasonal-naive plus day-of-week-adjusted moving average, with prediction intervals); capacity planning (ALOS 4.2 days, 90% occupancy ceiling, capacities A=40, B=28, C=22); and an independent reviewer (at least 3 weaknesses). The orchestrator merges everything into a one-page brief (about 300 words or fewer) plus an appendix, reconciling disagreements between the two methods.

Success criteria:

1. The cleaning step handles all three quirks, documents each fix, and leaves no negative or duplicate rows.
2. Both forecasting methods give a 7-day forecast with prediction intervals for all 3 hospitals. The brief states, per hospital, where the methods differ by more than 10% and how that was reconciled.
3. The capacity analysis applies LOS 4.2 and the 90% ceiling, states effective limits (A=36, B=25.2, C=19.8) and correctly flags at-risk hospitals.
4. The reviewer section lists at least 3 distinct methodological weaknesses and cites at least one concrete inconsistency, or states explicitly that none was found after checking.
5. The final deliverable has a one-page executive brief (about 300 words or fewer) plus a separate appendix with code and assumptions, and the brief's figures match the appendix.

Timing comparison

Approach	Duration (ms)
Single call	88862
Decomposed	197696

Decomposed program final status: pass

Decomposition attempts

Attempt 1: source generation = empty (writer returned null/whitespace); typecheck = skipped; execute = skipped. No error text. Fix in next attempt: asked the writer to generate the complete program from scratch, noting there was no source to fix.

Attempt 2: source generation = ok; typecheck = passed; execute = passed. No error text.

Grading

Single-call score: 8. Decomposed score: 2. Winner: single-call.

Rationale (verbatim):

The single-call solution delivers nearly every success criterion. It builds a synthetic dataset with the three quirks and handles each one: it replaces A's negative value, interpolates B's four missing days, and deduplicates C's repeated row. It gives SN and MA forecasts with prediction intervals for all three hospitals. It reports the per-hospital differences of more than 10% (B on every day) and reconciles them. It states the effective limits of 36, 25.2 and 19.8 beds, applies the 4.2-day length of stay, and flags B as at risk. The reviewer section lists several weaknesses and notes one logical inconsistency. The brief is about 300 words, with a separate appendix containing the code. Weaknesses: the numbers were hand-computed and the code was never run. The wording on the length-of-stay weakness is slightly muddled. The interpolation log shows B's day 9 as 5 and the others as 4.5, which is questionable. The reconciliation by averaging is simple.

The decomposed solution is largely incomplete. Its forecasting phase produced nothing, so it has no forecasts, no prediction intervals, no method comparison, no at-risk flags and no reconciliation. It does cover the cleaning rules and the effective limits, and its reviewer section is thorough. It also reports a scale mismatch, because its synthetic data averages about 100 admissions per day. But it does not fulfil the core deliverables.

Single-call solution

Single-call solution (condensed — full text omitted for size)

The solver built a synthetic dataset with the quirks (A day 31 = -7; B days 9, 26, 41, 56 missing; C day 45 duplicated), cleaned it to 252 rows, produced SN and 4-week MA forecasts with Poisson-based 95% PIs, found B differs by more than 10% every day (surge in week 12), reconciled by averaging, applied effective limits A=36, B=25.2, C=19.8, flagged B AT RISK (reconciled peak 26.25 beds vs 25.2), A on watch (33.6/36), C OK (16.8/19.8), and included a Python script plus a reviewer report with six weaknesses. The numbers were hand-computed, not executed. (The complete verbatim text could not be embedded in this issue body.)

Decomposed rig program source

import { agent, workflow, s } from "rig";

// Agent role: data-cleaning engineer who writes a Python script and documents repairs
const cleaner = agent({
  model: "medium",
  instructions:
    "You are a data-cleaning engineer. Given the task, generate a deterministic synthetic dataset (12 weeks of daily admissions for hospitals A, B, C) with the quirks: B missing 4 days, C one duplicated date row, A one value of -7. Write a Python script (pandas) that validates and repairs the data: interpolate B's 4 missing days linearly and flag them, drop the duplicate C row, replace A's negative value with interpolation from neighbours and flag it. Document each rule and each repair, and show the cleaned table summary (row counts per hospital, no negatives, no duplicates). Output the script, the repair log and a compact cleaned-data summary as plain text.",
  output: s.string,
});

// Agent role: forecasting analyst implementing two methods with prediction intervals
const forecaster = agent({
  model: "large",
  instructions:
    "You are a forecasting analyst. Using the cleaning output provided, implement in Python (1) a 7-day seasonal-naive baseline and (2) a day-of-week-adjusted moving average. Produce a 7-day forecast per hospital (A, B, C) for both methods with simple prediction intervals (residual std based, ~95%). Report the numbers in a table. State per hospital the mean and peak forecast of each method, and where the two methods differ by more than 10%. Output code plus numeric tables as plain text.",
  output: s.string,
});

// Agent role: capacity planner converting forecasts to bed and staffing recommendations
const planner = agent({
  model: "medium",
  instructions:
    "You are a hospital capacity planner. Using the forecasts provided, apply average length of stay 4.2 days and a 90% occupancy ceiling. State effective capacity limits: A=40*0.9=36, B=28*0.9=25.2, C=22*0.9=19.8 beds. Convert admissions to expected occupied beds (admissions per day x 4.2), flag each hospital whose forecast peak exceeds the effective limit, and give staffing and bed recommendations. Show your calculations explicitly. Output plain text.",
  output: s.string,
});

// Agent role: independent methodology reviewer
const reviewer = agent({
  model: "large",
  instructions:
    "You are an independent methodological reviewer. Audit the cleaning code, forecasting code, capacity assumptions and numbers supplied. List at least 3 distinct methodological weaknesses (small sample of 12 weeks, ignoring holidays, length-of-stay assumption, interpolation bias, interval assumptions, admissions vs. occupancy confusion). Check internal consistency across agents' numbers and cite at least one concrete numeric or logical inconsistency, or explicitly state that none was found after checking. Output plain text.",
  output: s.string,
});

// Agent role: executive writer who merges everything into brief plus appendix
const writer = agent({
  model: "medium",
  instructions:
    "You are an executive writer. Merge the inputs into: (1) a one-page EXECUTIVE BRIEF of at most 300 words which, per hospital, states where the two forecasting methods differ by more than 10% and how it was reconciled, states effective capacity limits (A=36, B=25.2, C=19.8) and which hospitals are at risk, and summarises reviewer caveats; (2) a separate APPENDIX containing the cleaning code and repairs, forecasting code and tables, capacity calculations, assumptions and the reviewer findings. All figures in the brief must match the appendix exactly. Output plain text with clear headings.",
  output: s.string,
});

// Workflow role: orchestrates the four phases and combines into the final deliverable
export default workflow({
  meta: {
    name: "hospital-bed-forecast",
    description: "Clean, forecast, plan capacity, review, and merge into an executive brief and appendix",
  },
  body: async ({ call }) => {
    const cleaned = await call(
      cleaner,
      "Produce the cleaning script, repair log and cleaned table summary for the hospital admissions dataset."
    );
    const forecasts = await call(
      forecaster,
      "Cleaning output:\n" + (cleaned ?? "") + "\n\nProduce both forecasting methods for all 3 hospitals with prediction intervals."
    );
    const plan = await call(
      planner,
      "Forecast output:\n" + (forecasts ?? "") + "\n\nProduce the capacity analysis and recommendations."
    );
    const review = await call(
      reviewer,
      "CLEANING:\n" + (cleaned ?? "") + "\n\nFORECASTS:\n" + (forecasts ?? "") + "\n\nCAPACITY:\n" + (plan ?? "") + "\n\nAudit all of this."
    );
    const final = await call(
      writer,
      "CLEANING:\n" + (cleaned ?? "") + "\n\nFORECASTS:\n" + (forecasts ?? "") + "\n\nCAPACITY:\n" + (plan ?? "") + "\n\nREVIEW:\n" + (review ?? "") + "\n\nWrite the brief and appendix."
    );
    return { solution: final ?? "" };
  },
});

Decomposed solution

Decomposed solution (condensed — full text omitted for size)

The final deliverable opens with "STATUS: INCOMPLETE. No forecast was supplied. The FORECASTS section is empty." The forecaster agent evidently returned null at runtime, so there are no forecasts, no prediction intervals, no method comparison and no at-risk flags. It does include the cleaning rules and script (raw rows A=84, B=80, C=85; cleaned 84 each), effective limits A=36, B=25.2, C=19.8, and a reviewer section noting a scale mismatch (synthetic data averages about 100/80/60 admissions per day against limits reached at 8.57/6.00/4.71) plus several methodological weaknesses. (The complete verbatim text could not be embedded in this issue body.)

Benchmark program

The benchmark program is a rig workflow() that picks a task with a small agent (with repair(), maxTurns: 2), solves it with a single-turn medium agent, has a medium agent write a decomposed rig program (2 attempts, typecheck then --server run via the rig CLI, with a 25-minute self-imposed deadline: 2 min per typecheck, 5 min per execution, clamped to the remaining budget), then grades both with a single-turn large agent. It imports only from "rig" and uses copilotEngine({ server: true }). (The complete source is not embedded here due to size.)

Verdict

Winner: single-call. The decomposed pipeline's forecasting step returned nothing, leaving forecasts, intervals and risk flags missing, while the single call covered nearly every criterion.

Note: the issue body was condensed because the complete report (about 40 KB) could not be submitted. The solution texts and the benchmark source are omitted or summarized here, which deviates from the verbatim requirement.

Generated by Daily Rig Decomposition Benchmark · copilot · auto · ~729 AIC · ⌖ ~1 AIC · ⊞ 4K · ◷

• expires on Oct 30, 2026, 1:51 PM UTC