[rig-decomposition-bench] Daily rig decomposition benchmark — 2026-10-10 — single-call

skipped: no valid source generated

skipped: no valid source generated

skipped: no valid source generated

skipped: no valid source generated

Vendor Incident Postmortem & Migration Brief

Incident: PayFlux Payment Processor Outage
Date/Time of Incident: 9:14 AM – 10:01 AM (47 minutes)
Prepared by: On-Call Operations Lead
Distribution: Leadership — Decision Required by End of Day

1. Root-Cause Summary

This incident had two distinct, non-overlapping contributing causes — one external, one internal. They must not be conflated, as they require different owners and different fixes.

External Fault (PayFlux):
Per PayFlux's own status page, the outage was caused by "a database failover that did not complete cleanly during a routine maintenance window." This is a vendor-side infrastructure failure, entirely outside our control, and is the root trigger of the incident. PayFlux is accountable for the initial 47-minute service disruption.

Internal Fault (Our Checkout Service):
Independent of PayFlux's failure, our own checkout service retried failed payment calls 5 times with no backoff logic. This retry storm amplified load on PayFlux's already-degraded systems and is estimated to have extended our own recovery time by approximately 12 minutes beyond PayFlux's actual service restoration. This is a self-inflicted aggravating factor, not caused by PayFlux, and is fully within our control to fix.

Summary statement: PayFlux caused the outage; our retry logic made our recovery worse and longer than it needed to be. Both facts are true simultaneously and must be addressed separately — one through vendor accountability (SLA credit), the other through engineering remediation (retry/backoff fix).

2. Financial Impact Tally

Note on the SLA credit: The $4,200 is a credit owed to us by PayFlux, meaning our net realized loss, after the credit is applied, is $168,080 ($172,280 − $4,200). However, for the purposes of this brief's total impact tally as requested, the full $172,280 figure represents combined lost revenue, support cost, and contractual credit value in play.

3. Immediate Remediation Plan — Retry-Logic Fix

Scope of work: Eliminate the no-backoff 5x immediate retry behavior in the checkout service and replace it with an exponential backoff (or capped retry) strategy to prevent load amplification during future vendor-side incidents.

Resources available: 3 engineers, today.
Estimated effort: 4 hours total.

Same-day timeline (single-day completion, parallelized where possible):

Total elapsed: ~5 hours wall-clock (11:00 AM–4:00 PM), with 4 engineer-hours of actual work distributed across the 3 engineers in parallel. Fix is live in production well before end of day.

4. Failover Processor Recommendation

Three options were evaluated using only the given facts:

Option A — Stick with PayFlux only:
No added cost, but no protection against recurrence. Given this is at least one known failover-related incident, relying solely on PayFlux leaves checkout fully exposed to any future maintenance or failover failure on their end.

Option B — Add ClearPay as secondary/failover:
ClearPay is 0.1% cheaper per transaction, which is attractive on cost. However, ClearPay has had 3 outages exceeding 15 minutes in the past 12 months — a materially worse reliability record than PayFlux has demonstrated in this single incident. Adding a failover processor that is itself failure-prone undermines the purpose of failover (redundancy against downtime), and could introduce a secondary single point of failure almost as risky as the primary.

Option C — Add SecurePay as secondary/failover:
SecurePay costs 0.3% more per transaction, which is a real and recurring cost. However, it has zero outages exceeding 15 minutes in the past 12 months, making it the most reliable of the three options by a clear margin. The tradeoff is a 3-business-day onboarding delay before it can serve as primary — meaning it cannot help today, but it can be onboarded this week as a standing failover for future incidents.

Recommendation: Add SecurePay as the automatic failover processor.
Reasoning: The cost premium (0.3% per transaction) is small and predictable, while a single 47-minute outage already cost $172,280 in combined impact — a single prevented incident of similar scale would far outweigh months of the fee differential. ClearPay's cheaper rate is not worth adopting for failover purposes given its 3 outages >15 minutes in 12 months, which is worse reliability than the system it would be backing up. SecurePay's 3-day onboarding delay is a one-time cost that should begin immediately in parallel with today's retry-logic fix, so that by the time onboarding completes, we have both the internal amplification bug fixed and a reliable external failover in place.

Action: Begin SecurePay onboarding today in parallel with the engineering fix; do not pursue ClearPay as a failover option.

No decomposed solution was produced — the decomposition program never executed successfully.

import { agent, configureAgent, copilotEngine, repair, s, workflow } from "rig";
import { execFileSync } from "node:child_process";
import { readFileSync } from "node:fs";

configureAgent(copilotEngine({ server: true }));

// ---- Wall clock budget ----
const START_MS = Date.now();
const DEADLINE_MS = START_MS + 25 * 60_000;
function remainingMs(): number {
  return DEADLINE_MS - Date.now();
}
function clamp(ms: number, min: number, max: number): number {
  return Math.max(min, Math.min(max, ms));
}

const SKILL_DIR = ".github/skills/rig";

type CliResult = { status: "ok" | "error" | "skipped"; stdout: string; stderr: string };

function runRigCli(source: string, args: string[], timeoutMs: number): CliResult {
  if (timeoutMs <= 0) {
    return { status: "skipped", stdout: "", stderr: "skipped: no time budget remaining" };
  }
  try {
    const stdout = execFileSync("node", ["rig.ts", ...args], {
      cwd: SKILL_DIR,
      input: source,
      timeout: timeoutMs,
      maxBuffer: 64 * 1024 * 1024,
      encoding: "utf8",
    });
    return { status: "ok", stdout, stderr: "" };
  } catch (err: any) {
    const stdout = typeof err?.stdout === "string" ? err.stdout : (err?.stdout?.toString?.() ?? "");
    const stderr = typeof err?.stderr === "string" ? err.stderr : (err?.stderr?.toString?.() ?? String(err?.message ?? err));
    return { status: "error", stdout, stderr: stderr || String(err) };
  }
}

// ---- Task picker ----
const TaskSchema = s.object({
  title: s.string,
  domain: s.string,
  description: s.string,
  successCriteria: s.nonEmptyArray(s.string),
});

// Agent role: pick one concrete, complicated single-day task suited to multi-agent decomposition.
const taskPicker = agent({
  name: "taskPicker",
  model: "small",
  maxTurns: 2,
  addons: [repair()],
  instructions:
    "Pick one concrete, complicated task that a person or team might face in a single day, " +
    "and that would naturally benefit from being split across sub-agents running different models " +
    "(e.g. multi-source research and synthesis, a multi-file coding task with distinct design/implementation/review phases, " +
    "a structured report combining several independent analyses, or a multi-step data transformation pipeline). " +
    "The task must be fully self-contained and solvable purely from its own written description text " +
    "(no external files, no live web access, no repository access). It must be concrete enough that a grader " +
    "can check whether a written solution satisfies each success criterion just by reading the solution text. " +
    "Prefer variety across domains (software, writing, research, data, operations, planning, etc.) rather than always picking coding. " +
    "Return a short title, a one-word-ish domain tag, a one-paragraph description containing every detail needed to solve the task " +
    "with no outside information, and 3 to 6 concrete, checkable success criteria.",
  output: TaskSchema,
});

// ---- Single-call solver ----
// Agent role: solve the whole task alone in a single turn, addressing every success criterion.
const singleCallSolver = agent({
  name: "singleCallSolver",
  model: "medium",
  maxTurns: 1,
  instructions:
    "You will be given a task description and its success criteria. Produce the complete solution to the task, " +
    "addressing every success criterion as thoroughly as you can in one response. Do not delegate, do not use tools, " +
    "just write out the full solution text (including Markdown code fences if appropriate for the content).",
  output: s.string,
});

// ---- Decomposition writer and fixer ----
const WRITER_SKELETON = `import { agent, workflow, s } from "rig";

// Agent role: <role>
const analyze = agent({ model: "small", instructions: "...", output: s.string });

// Workflow role: combines the agents' outputs into the final solution
export default workflow({
  meta: { name: "decomposed-solution", description: "..." },
  body: async ({ call }) => {
    const analysis = await call(analyze, "...");
    return { solution: analysis ?? "" };
  },
});
`;

const WRITER_CONTRACT = `The generated program must import only from "rig". Its root default export must take no
required input. The root must be either:
(a) a workflow() whose spec accepts only "meta" (with "name" and "description"), an optional "input",
    and "body" -- it has NO "output" and NO "execute" field. The final { solution } object returned by
    the workflow is exactly what "body" returns (a plain JS return statement, not any special field name
    beyond being an object with a "solution" string property).
(b) an agent() root that instead declares output: s.object({ solution: s.string }).
The program must split the task across AT LEAST TWO agents, each with a "// Agent role: ..." comment
above its declaration. Each agent should pick model: "small" for simple sub-steps and "medium" or "large"
for harder sub-steps. The agents must be coordinated (via a workflow() body, or via agents: {} delegation)
so that the FINAL answer COMBINES their outputs -- never have a single agent solve the whole task alone.
Use s.string for every agent whose result is free-form solution text; reserve object schemas for structured
metadata only, and never make a solver re-emit a large chunk of solution text wrapped inside an object field
other than the final top-level "solution" field.

Here is the REQUIRED root shape, abridged to a single agent for brevity -- your generated program must use
this exact contract for its root export, but must contain at least two agents performing distinct sub-steps:

\`\`\`ts
${WRITER_SKELETON}
\`\`\`

Return ONLY the complete TypeScript source code of the program. No prose, no explanation, no Markdown code
fences around it -- just the raw .ts source text.`;

// Agent role: write a rig program that decomposes the task across at least two sub-agents.
const decompositionWriter = agent({
  name: "decompositionWriter",
  model: "medium",
  maxTurns: 1,
  instructions:
    "You are writing a standalone rig TypeScript program (not using any tool or file access) that will be " +
    "typechecked and executed by an external CLI. " + WRITER_CONTRACT,
  output: s.string,
});

// Agent role: fix or regenerate the decomposition program given a captured error (or write it from scratch).
const decompositionFixer = agent({
  name: "decompositionFixer",
  model: "medium",
  maxTurns: 1,
  instructions:
    "You are revising (or, if told there is no prior source, writing from scratch) a standalone rig TypeScript " +
    "decomposition program. " + WRITER_CONTRACT +
    " If given prior source and an error, fix precisely what the error names while preserving everything that " +
    "already worked -- do not invent unrelated API edits of your own.",
  output: s.string,
});

type AttemptRecord = {
  attempt: number;
  sourceGeneration: "ok" | "empty" | "skipped";
  typecheck: "ok" | "error" | "skipped";
  typecheckOutput: string;
  execute: "ok" | "error" | "skipped";
  executeOutput: string;
  fixNote: string;
};

function isBlank(s: string | null | undefined): boolean {
  return s == null || s.trim().length === 0;
}

// ---- Grader ----
const GradeSchema = s.object({
  singleCallScore: s.number,
  decomposedScore: s.number,
  winner: s.enum("single-call", "decomposed", "tie"),
  rationale: s.string,
});

// Agent role: grade both solutions 0-10 against the success criteria, picking a winner.
const grader = agent({
  name: "grader",
  model: "large",
  maxTurns: 1,
  instructions:
    "You will be given a task description, its success criteria, and two candidate solutions: 'single-call' and " +
    "'decomposed'. Score each solution from 0 to 10 on how completely and correctly it satisfies the success " +
    "criteria. Judge ONLY correctness and completeness of content -- never on length, formatting verbosity, or " +
    "which approach (single-call vs decomposed) produced it. Pick a winner: 'single-call', 'decomposed', or 'tie'. " +
    "Explain your comparison, citing which criteria each solution met or missed.",
  output: GradeSchema,
});

// Workflow role: run the single-call vs decomposed-agent benchmark and return a full report.
export default workflow({
  meta: {
    name: "rig-decomposition-bench",
    description: "Benchmark single-call solving vs multi-agent decomposition on a generated task",
  },
  body: async ({ call }) => {
    // Step 1: pick a task.
    const task = (await call(taskPicker, "Pick one concrete, complicated single-day task.")) ?? {
      title: "Fallback task (task picker failed)",
      domain: "unknown",
      description: "The task picker agent failed to return a usable task.",
      successCriteria: ["N/A"],
    };

    const criteriaText = task.successCriteria.map((c, i) => `${i + 1}. ${c}`).join("\n");
    const taskPrompt =
      `Task: ${task.title}\nDomain: ${task.domain}\n\nDescription:\n${task.description}\n\n` +
      `Success criteria:\n${criteriaText}`;

    // Step 2: single-call solve.
    let singleCallSolution = "";
    const singleStart = Date.now();
    if (remainingMs() > 20_000) {
      singleCallSolution = (await call(singleCallSolver, taskPrompt)) ?? "";
    }
    const singleCallDurationMs = Date.now() - singleStart;

    // Step 3: decomposition.
    const decompStart = Date.now();
    const attempts: AttemptRecord[] = [];
    let decomposedSource = "";
    let decomposedSolution = "";
    let decomposedPassed = false;

    const MAX_ATTEMPTS = 2;
    for (let attemptNum = 1; attemptNum <= MAX_ATTEMPTS; attemptNum++) {
      if (remainingMs() < 60_000) {
        attempts.push({
          attempt: attemptNum,
          sourceGeneration: "skipped",
          typecheck: "skipped",
          typecheckOutput: "skipped: insufficient time budget remaining",
          execute: "skipped",
          executeOutput: "skipped: insufficient time budget remaining",
          fixNote: "",
        });
        continue;
      }

      let generated = "";
      let sourceGeneration: "ok" | "empty" = "empty";
      let fixNote = "";

      if (attemptNum === 1) {
        generated = (await call(decompositionWriter, taskPrompt)) ?? "";
      } else {
        const prevAttempt = attempts[attempts.length - 1];
        if (isBlank(decomposedSource)) {
          const fixPrompt =
            `${taskPrompt}\n\nThere is no prior source to fix -- the previous generation attempt returned an ` +
            `empty result. Generate the complete program from scratch using the original task and the required skeleton.`;
          generated = (await call(decompositionFixer, fixPrompt)) ?? "";
          fixNote = "Previous attempt produced no source; regenerated from scratch.";
        } else {
          const errText =
            prevAttempt?.execute === "error"
              ? prevAttempt.executeOutput
              : prevAttempt?.typecheck === "error"
                ? prevAttempt.typecheckOutput
                : "unknown failure";
          const fixPrompt =
            `${taskPrompt}\n\nHere is the prior program source:\n\n\`\`\`ts\n${decomposedSource}\n\`\`\`\n\n` +
            `Running it produced this exact error:\n\n${errText}\n\n` +
            `Fix precisely what this error names while preserving everything that already worked.`;
          generated = (await call(decompositionFixer, fixPrompt)) ?? "";
          fixNote = `Fed prior source plus captured error back to the fixer.`;
        }
      }

      if (isBlank(generated)) {
        attempts.push({
          attempt: attemptNum,
          sourceGeneration: "empty",
          typecheck: "skipped",
          typecheckOutput: "skipped: no valid source generated",
          execute: "skipped",
          executeOutput: "skipped: no valid source generated",
          fixNote,
        });
        continue;
      }

      sourceGeneration = "ok";
      decomposedSource = generated;

      // Typecheck.
      const tcTimeout = clamp(remainingMs(), 0, 2 * 60_000);
      const tcResult = runRigCli(generated, ["--typecheck"], tcTimeout);
      const typecheck: "ok" | "error" | "skipped" =
        tcResult.status === "skipped" ? "skipped" : tcResult.status === "ok" ? "ok" : "error";
      const typecheckOutput =
        tcResult.status === "skipped" ? tcResult.stderr : (tcResult.stdout + tcResult.stderr).trim();

      let execute: "ok" | "error" | "skipped" = "skipped";
      let executeOutput = "skipped: typecheck did not pass";

      if (typecheck === "ok") {
        const exTimeout = clamp(remainingMs(), 0, 5 * 60_000);
        const exResult = runRigCli(generated, ["--server"], exTimeout);
        execute = exResult.status === "skipped" ? "skipped" : exResult.status === "ok" ? "ok" : "error";
        executeOutput =
          exResult.status === "skipped" ? exResult.stderr : (exResult.stdout + exResult.stderr).trim();

        if (execute === "ok") {
          try {
            const parsed = JSON.parse(exResult.stdout);
            const sol = parsed?.solution;
            if (typeof sol === "string" && sol.trim().length > 0) {
              decomposedSolution = sol;
              decomposedPassed = true;
            } else {
              execute = "error";
              executeOutput = `Execution produced JSON without a non-empty string "solution" field. Raw stdout:\n${exResult.stdout}`;
            }
          } catch {
            execute = "error";
            executeOutput = `Execution stdout was not valid JSON. Raw stdout:\n${exResult.stdout}\nStderr:\n${exResult.stderr}`;
          }
        }
      }

      attempts.push({
        attempt: attemptNum,
        sourceGeneration,
        typecheck,
        typecheckOutput,
        execute,
        executeOutput,
        fixNote,
      });

      if (decomposedPassed) break;
      if (remainingMs() < 60_000) break;
    }

    const decomposedDurationMs = Date.now() - decompStart;

    // Step 4: grade.
    const decomposedForGrading = decomposedPassed
      ? decomposedSolution
      : "(No valid decomposed solution was produced -- all attempts failed to generate, typecheck, or execute successfully.)";

    let grade: {
      singleCallScore: number;
      decomposedScore: number;
      winner: "single-call" | "decomposed" | "tie";
      rationale: string;
    } = {
      singleCallScore: 0,
      decomposedScore: 0,
      winner: "single-call",
      rationale: "Grading was skipped due to insufficient remaining time budget.",
    };

    if (remainingMs() > 20_000) {
      const gradePrompt =
        `${taskPrompt}\n\n--- Solution A (single-call) ---\n${singleCallSolution || "(empty)"}\n\n` +
        `--- Solution B (decomposed) ---\n${decomposedForGrading}\n\n` +
        `Score Solution A as singleCallScore and Solution B as decomposedScore.`;
      grade = (await call(grader, gradePrompt)) ?? grade;
    }

    let benchProgramSource = "";
    try {
      benchProgramSource = readFileSync("/tmp/gh-aw/agent/bench.ts", "utf8");
    } catch {
      benchProgramSource = "(unable to read own source file at /tmp/gh-aw/agent/bench.ts)";
    }

    return {
      task,
      singleCallDurationMs,
      decomposedDurationMs,
      singleCallSolution,
      decomposedSolution: decomposedPassed ? decomposedSolution : "",
      decomposedProgramSource: decomposedSource,
      attempts,
      decomposedPassed,
      grade,
      benchProgramSource,
    };
  },
});