[rig-tasks] Daily rig evaluation — 2026-10-13 — 10/10 passed

This is an automated daily evaluation report, not a change request, and the body names no single file to edit. The documentation gaps it raises point at SKILL.md (guidance on choosing defineTool over prose for arithmetic) and at a possible references page on maxTurns partiality, citing sample 552. Done would mean a maintainer has triaged which of those suggestions become concrete work, so a newcomer should look for a specific follow-up issue rather than start here.

Summary

Theme seed: 649507a8347906fbdfc3cd73c78f23c0.

All 10 seeded domain/artifact/constraint combinations were accepted (one task per theme, no duplicates, no unused themes):

Theme	Outcome
language learning / nested catalogs / partial observations	accepted — 551
weather and climate / resource schedules / partial observations	accepted — 552
travel planning / linked records / bounded resource budgets	accepted — 553
food and recipes / event timelines / conflicting sources	accepted — 554
scientific experiments / linked records / partial observations	accepted — 555
libraries and archives / tabular measurements / conflicting sources	accepted — 556
energy usage / event timelines / conflicting sources	accepted — 557
community events / nested catalogs / bounded resource budgets	accepted — 558
media production / linked records / multilingual labels	accepted — 559
education / linked records / conflicting sources	accepted — 560

The cached task-pool backlog (/tmp/gh-aw/cache-memory/task-pool.json) contained 28 entries, all already evaluated (18 duplicate, 10 published); none were eligible for reuse, so all 10 selection slots went to freshly seeded themes.

Task	Description	Typecheck	Novelty / closest sample	Key finding
1 (new)	nested-catalog-partial-rollup — recursive completion roll-up over an arbitrary-depth nested catalog, distinguishing missing-data from zero-progress	✅ pass	new lesson vs. 512-nested-tsconfig-validator.md / 377-toml-config-extractor.md / 543-transit-stop-translation-joiner.md	Expander correctly used defineTool for the recursive aggregation logic (pure TS) rather than trying to push recursion into the prompt; clean separation of deterministic math from the agent call.
2 (new)	resource-schedule-partial-lookup — defineTool lookup table bounded by an explicit maxTurns budget, leaving some stations unresolved purely due to budget	✅ pass	new lesson vs. 549-shot-log-gap-reconciler.md / 541-station-report-budget.md / 543-transit-stop-translation-joiner.md	Model correctly distinguished "unresolved due to budget" from "not found in table" by instructing the agent to query in strict order and stop once turns run out — relies entirely on prompt discipline since maxTurns is enforced by the harness, not the model.
3 (new)	linked-record-chain-budget-walk — two-phase workflow() resolving a nextId-linked record chain, then a budget phase cutting it off	✅ pass	new lesson vs. 546-itinerary-budget-pipeline.md / 320-budget-aware-crawler.md / 548-warehouse-slot-allocator.md	Clean use of call.json per chain hop with a while (currentId) loop, then a separate deterministic TS loop for the budget cutoff — good separation of model-driven resolution vs. deterministic budget logic.
4 (new)	conflicting-source-timeline-reconciler — median + cookbook-fallback tie-break across 3 disagreeing numeric sources	✅ pass	new lesson vs. 218-ts-interface-conflict-checker.md / 380-npm-peer-dep-conflict-checker.md / 550-venue-booking-diff-writer.md	The tie-break rule (evenly-spaced values → fall back to cookbook) was expressed entirely in prose for the model to apply; this is inherently fuzzier than a defineTool-computed median, a design trade-off worth flagging (see below).
5 (new)	linked-record-foreign-key-join — two-table FK join via defineTool, with dangling-reference handling	✅ pass	new lesson vs. 543-transit-stop-translation-joiner.md / 549-shot-log-gap-reconciler.md / 08-extract-package-scripts.md	Tool cleanly encodes both lookup tables and the join logic in deterministic TS; the nullable adjustedValue/adjustmentReason pair is a reusable pattern for "value or reason it's missing."
6 (new)	tabular-measurement-source-priority-reconciler — asymmetric priority-override rule with 4-way s.enum resolution-path classification	✅ pass	new lesson vs. 545-recipe-conversion-delta.md / 207-config-drift-reconciler.md / 86-npm-audit-simplifier.md	The "zero digital count overridden by nonzero shelf count" edge case required careful prose to avoid ambiguity between "agreed" and "overridden" — the model handled it correctly on the first pass.
7 (new)	parallel-timeline-conflict-flagger — Promise.all fan-out over two overlapping timelines, aggregator explicitly flags (not merges) timestamp disagreements	✅ pass	new lesson vs. 550-venue-booking-diff-writer.md / 320-budget-aware-crawler.md / 529-parallel-wordfreq-aggregator.md	Straightforward reuse of the established Promise.all fan-out pattern from 529, but the reconciliation logic (agree/flag/passthrough) is deterministic TS in the aggregator phase rather than another model call — good choice given it's pure set logic.
8 (new)	nested-catalog-budget-allocator — top-down volunteer-hour budget across a 3-level festival→stage→act catalog with subtree cutoff	✅ pass	new lesson vs. 548-warehouse-slot-allocator.md / 541-station-report-budget.md / 546-itinerary-budget-pipeline.md	Entire allocation algorithm (proportional share, running-total cutoff, subtree-unfunding) is expressed in prose for the model to execute — this is the most arithmetic-heavy prompt in the batch and a candidate for a defineTool to guarantee correctness (see below).
9 (new)	multilingual-linked-record-manifest — defineTool resolving a set of locales per shot→scene linked record, tracking per-entity missing locales	✅ pass	new lesson vs. 543-transit-stop-translation-joiner.md / 547-lab-result-steered-translation.md / 549-shot-log-gap-reconciler.md	Clean extension of the single-locale lookup pattern (543) to a locale array input with a missingLocales array output — a good template for future multi-key lookup tools.
10 (new)	linked-record-recency-reconciler-workflow — two-phase workflow: extract conflicting linked records, then recency+null-fallback reconcile with p.write audit trail	✅ pass	new lesson vs. 207-config-drift-reconciler.md / 550-venue-booking-diff-writer.md / 546-itinerary-budget-pipeline.md	Combines extraction/reconciliation phase split with a p.write side effect in the second phase — a reusable pattern for "resolve, then persist an audit trail" workflows.

---

Problems encountered

No typecheck failures this run — all 10 generated programs passed --typecheck on the first attempt.

One environment/infrastructure problem was encountered and resolved during setup, unrelated to the rig codebase itself:

• npm ci initially failed repeatedly with DEPTH_ZERO_SELF_SIGNED_CERT errors against the ms-feed-25.pkgs.visualstudio.com 1ES proxy feed referenced in the committed package-lock.json's resolved URLs, and separately with npm 10.9.9's "Exit handler never called!" bug. Root cause: the corporate squid proxy in this sandbox could reach registry.npmjs.org directly (confirmed via curl) but not the 1ES feed host. Fix: locally patched package-lock.json to point resolved URLs at registry.npmjs.org instead of the feed, ran npm ci successfully, then restored the original (feed-pointing) package-lock.json before committing — no lockfile changes are included in the PR.

Separately, the task-generator and rig-expander subagents' default ("small"/"large") model routing failed consistently with 400 Cannot translate Copilot request feature 'frequency_penalty' between Responses and Chat Completions. Routing model "gpt-5.4-mini"/"gpt-5.6-luna" to /responses is incompatible.... This is an infra routing issue, not a rig API problem. Both subagents succeeded immediately after retrying with an explicit claude-sonnet-5 model override.

---

Improvement opportunities

Missing or undiscoverable schema helpers (s.*)

No gaps surfaced this run. s.nullable, s.enum, s.record, and s.path all covered the needed shapes directly (tasks 1, 5, 6, 9, 10).

Missing or undiscoverable prompt helpers (p.*)

No gaps. p.bash heredocs comfortably embedded JSON, CSV, and line-oriented fixture data across all 10 tasks; p.write handled the one audit-trail side effect (task 10) without friction.

Error message quality

Not exercised this run — no typecheck failures occurred to evaluate error message clarity.

API ergonomics

• Two of the ten tasks (6 and especially 8) embed a non-trivial deterministic algorithm (asymmetric priority-override, proportional top-down budget allocation with subtree cutoff) entirely in prose for the model to execute, rather than offloading it to a defineTool. This is a modeling choice available today, but SKILL.md could more strongly nudge "if the task is arithmetic/deterministic, prefer defineTool over prose" — the expander defaulted to prose-only for arithmetic-heavy tasks even though defineTool was available and used correctly elsewhere in the same batch (tasks 1, 2, 5, 9).
• Task 2's budget semantics rely entirely on the agent self-regulating against maxTurns via prompt instructions ("you will run out before you can query every station — stop when you run out"); there's no first-class way for the program itself to detect "the budget was hit" versus "the model just decided to stop early" other than trusting the model's own narration. A documented pattern (or harness-level turn-count introspection accessible to the prompt) for "tell me how many turns you have left" would make this lesson more robust and less reliant on the model accurately self-reporting its own exhaustion.

Candidate lint rules

No new lint rule is proposed this run — all patterns observed (nested defineTool recursion, maxTurns + tool budget, chain walks via call.json, Promise.all fan-out aggregation, s.enum resolution classifications) are either already idiomatic or are one-off prompt-engineering choices rather than repeated structural mistakes that linting could catch.

Documentation gaps

• SKILL.md's guidance on choosing defineTool vs. prose-only arithmetic could be more explicit (see API ergonomics above) — currently the decision seems to default to the expander's judgment per-task rather than a documented rule of thumb.
• No example in the reference docs currently demonstrates the "maxTurns creates a genuinely partial result set, distinct from in-table absence" pattern now captured in sample 552 — worth considering for a future references page on invocation-option-driven partiality, alongside the existing timeout() addon guidance.

---

Tasks run today

• (new) nested-catalog-partial-rollup — bucket schemas, family nested-catalog-partial-rollup, lesson: recursive roll-up over variable-depth nested catalog distinguishing missing vs. zero, closest samples 512/377/543, novelty: neither validates+aggregates nor propagates null-for-unknown through parent levels
• (new) resource-schedule-partial-lookup — bucket tools, family resource-schedule-partial-lookup, lesson: defineTool + maxTurns budget exhaustion leaving entries unresolved distinct from not-found, closest samples 549/541/543, novelty: neither caps tool-call turns to produce a budget-exhaustion-specific unresolved list
• (new) linked-record-chain-budget-walk — bucket workflows, family linked-record-chain-budget-walk, lesson: linked-chain walk with cumulative cost cutoff reporting the cut point, closest samples 546/320/548, novelty: neither walks a nextId-linked chain with a dollar-cost budget reporting the specific cut record
• (new) conflicting-source-timeline-reconciler — bucket schemas, family conflicting-source-timeline-reconciler, lesson: 3-source median/tie-break reconciliation into one resolved value per step, closest samples 218/380/550, novelty: neither resolves 3 disagreeing numeric sources via a deterministic tie-break, only flags/diffs
• (new) linked-record-foreign-key-join — bucket tools, family linked-record-foreign-key-join, lesson: two-table FK join via one defineTool with dangling-reference handling, closest samples 543/549/08, novelty: neither joins two separate cross-referencing tables (vs. one flat lookup table)
• (new) tabular-measurement-source-priority-reconciler — bucket schemas, family tabular-measurement-source-priority-reconciler, lesson: asymmetric priority-override rule with per-row enum classification across two full conflicting tables, closest samples 545/207/86, novelty: neither applies an override rule or classifies resolution path via enum
• (new) parallel-timeline-conflict-flagger — bucket workflows, family parallel-timeline-conflict-flagger, lesson: parallel fan-out over two overlapping timelines, explicit disagreement flagging instead of merging, closest samples 550/320/529, novelty: no parallel fan-in workflow sample flags (vs. merges/dedupes) overlapping-key disagreement
• (new) nested-catalog-budget-allocator — bucket schemas, family nested-catalog-budget-allocator, lesson: top-down budget allocation through a 3-level nested catalog with subtree-level cutoff, closest samples 548/541/546, novelty: neither allocates through nested (vs. flat) structure with cutoff propagating to whole subtrees
• (new) multilingual-linked-record-manifest — bucket tools, family multilingual-linked-record-manifest, lesson: multi-locale simultaneous resolution per linked record with per-entity missing-locale tracking, closest samples 543/547/549, novelty: 543 resolves only one fixed target locale per entity
• (new) linked-record-recency-reconciler-workflow — bucket workflows, family linked-record-recency-reconciler-workflow, lesson: two-phase extraction + recency/null-fallback reconciliation workflow with p.write audit trail, closest samples 207/550/546, novelty: neither is a multi-phase workflow with a recency+null-fallback rule and persisted audit trail

Generated by Daily Rig Task Generator · copilot · auto · 786.9 AIC · ⌖ 22.9 AIC · ⊞ 11.4K · ◷