Priority: P1 — fund-safety program: map every incident class against CDK's saga model; land safe wins pre-release, deep-dive after (not a release blocker)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Start with AGENTS.md, then read the referenced incident issues (#497, #498, #500) and the saga entry points named in cashubtc/cdk, especially crates/cdk/src/wallet/{swap,send,melt,issue,receive}/saga/ and crates/cdk/src/wallet/saga/mod.rs. Compare TollGate’s current recovery and transaction handling with CDK’s prepare, persistence, and resume model; the issue calls for a post-v0.6.0 deep dive and a separate list of strictly safer pre-release actions. Done means a complete incident-class map, comparison, and decision list—not implementing the structural fix.
Written by the indexing model from the issue text.
Description
Priority: P1 — the fund-safety program: every incident class mapped against CDK's saga model; land the safe wins before v0.6.0, run the deep dive after. Explicitly NOT a release blocker by itself — items that carry their own P0 keep it.
Framing
TollGate's incident history is one structural story told five ways: irreversible Cashu operations (swap/mint/melt) whose supporting state — derivation counters, secrets, produced tokens, business outcome — is not durable at the right moments, and whose ambiguous outcomes are retried instead of reconciled. AGENTS.md encodes the rules earned from these incidents; this issue turns them into a program: a complete map of what happened and why, a comparison against CDK's saga model (the reference design for crash-safe Cashu), a short pre-release decision list (only strictly-better-than-status-quo items), and a post-0.6.0 deep dive that designs the real fix instead of patching symptoms.
This issue does not lower anyone's priority: #497 keeps its P0; pin audits descending from #494–#496 stay mandatory before any tag. What it does say: the deep-dive work must not be rushed in under a release deadline — that is exactly how the half-fixes below were born.
The incident map
Class A — deterministic-derivation reuse (the 10002 brick family)
| Issue | What happened | Why it happened |
|---|---|---|
| #257 | Production v0.5.0: every payment fails outputs have already been signed before; wallet bricked |
Counter incremented only after a successful swap; a transient mint failure (timeout/5xx/429) left the counter put, so the retry re-derived the same blinded messages → NUT-02 error 10002. Latent in tests because mock mints don't fail |
| #266 | The fix: IncrementKeysetCounter() moved before the swap call (gonuts v0.7.4) |
Correct for the first attempt — but a patch on one path, not an invariant; see #495 |
| #480 | Any service restart permanently broke swaps for every mint with prior history (Duplicate outputs) |
Two URL spellings of one mint created two keyset buckets; on restart the in-memory map lost the alias, AddMint re-registered the second spelling, and a freshly-fetched keyset record (counter 0) overwrote the persisted higher counter — SaveKeyset had no monotonic guard. Re-derivation from counter 0 → brick |
| #496 | The #480 fix existed on a branch but was not in the pinned dependency — production stayed vulnerable | Pin discipline gap; lesson: the class (any writer that can regress the counter) needs an audit, not just the one writer that fired |
| #494 | Melt-path variant: change outputs sent to the mint before any counter increment; response lost → TollGate's retry (fresh invoice, fresh quote) re-derived the same range → 10002 | Same increment-after-success pattern as #257, on the melt path — the class was fixed where it burned first, not everywhere |
| #495 | After any 10002-retry the persisted counter sits below the ranges actually exposed; every later first attempt eats a guaranteed 10002 and only the regenerated retry works | The #266 fix covered the first attempt; swapWithRetry's regenerated retry range was never persisted. Masked (payments succeed) until the retry itself collides or a mint rate-limits the double request |
| #640 | The #535 conformance lane measured the swap-timeout-retry scenario re-exposing outputs on 2026-10-05 main — same class, alive post-#480-fixes | An ambiguous outcome (response dropped after the mint processed it) was retried blindly with regenerated outputs instead of reconciling proof state first. Issue since closed by the gonuts pin — the pre-release checklist re-verifies that |
Root cause, structural: in gonuts the derivation counter is keyset state nested under a mint-URL-named bbolt bucket, with no transaction spanning "reserve counter → POST → save proofs". Every writer of keyset metadata is a counter writer, and discipline is manual. The brick family is what happens each time a writer or a retry path breaks that discipline.
Class B — crash windows that destroy created value
| Issue | What happened | Why it happened |
|---|---|---|
| #497 (P0, open) | Swap secrets/blinding factors exist only in memory; process death after PostSwap succeeds but before SaveProofs makes the mint-issued outputs unrecoverable — customer's token gone, no local trace |
Nothing durable is written between the irreversible POST and the save. The exact gap CDK's wallet saga exists to close |
| #375 | wallet drain cashu aborted on the first per-mint failure, discarding tokens already produced by earlier successful mints; wallets with two spellings of one mint showed phantom balances |
No partial-result contract, no journal, non-canonical mint identity. Fixed by #433 (journal-before-next-mint, partial results, canonicalization) — the journal is a hand-rolled, one-purpose saga record |
Class C — the business transaction spanning wallet + memory + NDS
| Issue | What happened | Why it happened |
|---|---|---|
| #258 / #403 | Payment consumed (Receive + swap irreversible), then ndsctl auth fails → customer's value sits in the operator wallet, no session, no refund path; restoreSession rewinds only the in-memory session map |
Wallet-internal atomicity ≠ business-transaction atomicity. Receive → session → gate-open → response has no compensating action and no record; the refund-vs-late-grant decision was never made. #403 closed; #502 generalizes it into a TollGate-level transaction record |
| #498 (P1, open) | The ambiguous-outcome class at the TollGate layer: timeouts treated as failure, retries launched without reconciling what the mint already did | Same anti-pattern as #640 one layer up |
Class D — stuck value nobody sweeps
| Issue | What happened | Why it happened |
|---|---|---|
| #500 (P1, open) | Proofs entering pending (failed send, ambiguous melt, reseller hand-off) stay reserved forever: invisible in balance, reclaim machinery (ReclaimUnspentProofs, GetPendingMeltQuotes) exists in gonuts but the daemon never calls it |
Recovery was built as tools (nutw CLI, scripts/token-recovery), never wired into the service lifecycle |
| #423 | clientd burns tokens on rejected payments; no reclaim path | Same gap on the client side |
| #417 | Keyset final_expiry silently kills held balances (proven in cloud lab: Keyset has expired) |
Spec-correct mint behavior (NUT-02); wallets must rotate off inactive keysets — hygiene, not a crash bug, but the same "value strands silently" outcome |
Class E — failure-path liveliness
| Issue | What happened | Why it happened |
|---|---|---|
| #525 / #532 | A wedged mint (accepts TCP, never answers) parked the fee precheck for 5+ minutes with no deadline; goroutines stacked | The wallet client's retry ladder (30 s × 5 attempts × 2 endpoints) has no context/deadline. Fixed by budget-bounding the fetch; the same idiom now guards #549's checkstate (10 s budget — a dead mint still costs ~8 s on the retry ladder, so the budget is load-bearing) |
What CDK does differently (the saga model)
Verified against cashubtc/cdk main on 2026-10-07: the saga layout #497/AGENTS.md cite exists exactly as described — crates/cdk/src/wallet/{swap,send,melt,issue,receive}/saga/ plus shared crates/cdk/src/wallet/saga/mod.rs, mirrored on the mint side (crates/cdk/src/mint/{swap/swap_saga,melt/melt_saga}/). The design, from source:
- Type-state sagas — operation states are distinct types (
swap/saga/state.rs:Initial {operation_id, keyset_policy}→prepare()→Prepared {input_ys, pre_swap, saga}); invalid transitions do not compile. - Prepare persists the intent —
swap/saga/mod.rsreserves the input proofs, reserves the derivation-counter range in one atomic call (increment_keyset_counter;counter_start = counter_end − derived_secret_count), and persists the saga record (add_saga) for crash recovery — inputs, counter range and premint secrets are durable before the irreversible POST. Registered compensating actions revert the reservation and delete the saga on failure. - Compensating actions, LIFO —
wallet/saga/mod.rs: each step registers aCompensatingAction; rollback runs them in reverse order (errors logged, execution continues) — the compensation our Class C lacks. - Resume is replay-first —
receive/saga/resume.rs: forSwapRequested, replay the originalpost_swap(a NUT-19-cached mint response returns the signatures immediately); only on replay failure check whether inputs are spent and fall back to NUT-09/restore.ProofsPending→ compensate by removing pending proofs. Boot walks every persisted saga and resumes or compensates it. - Proof ownership markers — reserved proofs carry
used_by_operation: operation_id, and parent sagas (send/melt/receive) compose the swap saga withProofReservation::Skipso exactly one layer owns each proof lifecycle — the ownership-marker design #500 item 4 and #702 ask for, already shipped there. - Handed-out tokens are first-class — send outputs persist as
State::Reserved(not spent, not unspent — the mint decides);reclaim.rssync_proofs_statebatch-reconciles against NUT-07 and updates locally per verdict.
Their maintenance history is itself study material: cdk#1278 fixed a swap path missing its reconciliation wrapper (our #640 exactly); #1456 then removed the wrapper from receive ops (receive has no inputs of ours to reclaim) and #1457 slimmed the saga table; #1446 added mint-side compile-time row locks. Lesson: wrapper idioms rot when one call site is forgotten — the saga makes the safe path the only path that compiles.
One sentence: CDK persists the intent (inputs, counter range, blinded messages) before the irreversible call, marks ownership on every proof, and resumes by replay; gonuts persists only the counter and hopes the rest fits in memory between two lines of code. Every Class A and B incident lives in that gap. The structural fix direction is why AGENTS.md flags migration (#305/#277), with the MIPS/CGO_ENABLED=0 constraint pushing cdk-go behind the sidecar (src/tollwallet/sidecar.go) rather than in-process.
External study material — where to study what, and why
cashu-audit (ours — Amperstrand/cashu-audit)
The 3-layer framework (greatspectations spec-quote anchors → cross-implementation divergence database → AI audit prompts; #549's verbatim NUT quotes already come from this machinery). Fund-safety-relevant holdings:
issues/interrupted-operation-recovery.md(GH-tracked, HIGH, "biggest underspecified area"): recovery from interrupted operations is unspecified — every implementation solves it privately, so interoperable wallets cannot share recovery assumptions; proposes a NUT for operation-recovery semantics (state machines across crash points, idempotency keys, restore obligations). This issue's deep dive should feed that upstream draft — TollGate is the interoperable payment-router case study the gap hurts most.divergences/RESOLVED-ISSUE041-concurrent-double-melt.md(CRITICAL, resolved 2026-07-29): TOCTOU on the melt-quoteUNPAID→PENDINGtransition — read-check, manyawaitpoints, unconditional write 650 lines later; two concurrent melts interleaved and the loser's proofs were burned for nothing. Fix: atomic compare-and-set at the write site; the race loser is rejected before any proof is touched. Direct lesson for our #639 (concurrent duplicate token POST double-grants) and the #494 melt paths: CAS the state transition where it is enforced, don't check it earlier and write later.issues/keyset-final-expiry-enforcement.md: cdk enforcesfinal_expiryat swap (error 12003), nutshell never expires; inactive ≠ expired — inactive keysets MUST still accept swap inputs. Cross-checked against our cloud-lab rotation lane; downstream tracking = #417.divergences/CDK-VS-NUTSHELL-ALIGNMENT-2026-07-29.md: the triage method worth adopting (both references pass → your bug; references differ → genuine divergence; "default to CDK unless the NUT spec is explicit"), plus open items including proofs stuck PENDING ~30 min on payment-timeout deferral — our Class D manifesting in another implementation.LEARNINGS-FROM-ISSUES.md(patterns from 36+ fixed issues): five catch-patterns, notably "state-machine bugs only manifest with specific proof configurations → E2E tests with all state combinations" — the design brief for extending the #535 fault-injection lane.signoffs/gonuts-tollgate/(NUT-by-NUT audits of our fork, incl.POST-FIX-VERIFICATION-20260728): re-run these against the release pin as part of the pre-release pin audit; record outcomes back into cashu-audit.signoffs/comparisons/NUT-{05,07,08,09}-comparison.md: per-NUT wallet-relevant differences (melt change, checkstate shape, fees, restore).issues/nut05-blank-change-outputs.md(HIGH, production incident exists): melt/swap change outputs are blank — pairing is positional, declared amounts are ignored, take amounts from the returned signature. The wallet-side "I declared denominations" mental model is wrong; relevant to #494 (change-output handling) and #414 (input fees).
Conduition — NUT-13 deterministic-secrets disclosure (mandatory reading for Class A)
https://conduition.io/code/cashu-disclosure/ (2026-01, bounty-paid): NUT-13 secrets derived via BIP32 paths keyed on a 31-bit integer residue of the keyset ID let a malicious mint force cross-mint secret/preimage reuse, then use the target mint's NUT-09 /restore as an oracle to harvest blind signatures. Two consequences: the derivation counter is not just a fund-safety component but an attack surface ("wallets must manage the stateful counter correctly and verify keyset IDs"), and /restore — which recovery designs lean on — doubles as an attack oracle. Fixes: HMAC over the full keyset ID + counter (long-term), residue-collision guards (short-term), 256-bit keyset-ID v2 (protocol level). Most Cashu wallets were affected.
Nutshell (cashubtc/nutshell)
- Mint
Ledgermarks input proofs pending in a DB table as a distributed lock for the duration of processing — double-spend protection as a persisted state machine (witness fields valid only in SPENT, enforced at the model layer). - Wallet restore-from-mnemonic then
invalidate: reconcile restored proofs against NUT-07 checkstate before trusting them (cashu/wallet/wallet.py~L1120) — the same authority-ordering as #549; thekeysetstable keeps per-keyset derivation counters as first-class wallet state. - Proof selection filters to unreserved proofs on active keysets (
wallet.py~L1175) — rotation hygiene for #417; noted divergence: nutshell never expires keysets. - Far-field: nutshell#1046 Proof-of-Liabilities (MS-SMT epoch trees,
cashu pol auditletting wallets trustlessly audit spent/issued trees) — the "mint as authority" endgame.
Pre-v0.6.0 decision list (strictly better than status quo, small blast radius)
- Ship #549 — read-only liveness report; no wallet state touched; already battery-green.
- Pin audit (cheap, mandatory-feeling): verify the pinned gonuts tag carries the #494/#495/#496-class fixes and the #640 fix; re-run the #535 conformance fast-subset lane on the release candidate and attach the evidence to the release issue (#665/#339).
- #500 boot-time reconciliation — pre-release candidate only: bounded, operates within our own wallet, and turns stuck value into self-healing. Include if the conformance lane is green on it; otherwise it slips with the deep dive.
- Do NOT rush #497 (pre-persisted swap intents) pre-release. It is research-first by its own framing; every half-fix in Class A was a deadline patch.
- #502 (business-transaction record) stays post-release unless already stacked and reviewed.
- Informational read-audit of the NUT-13 derivation question (Conduition disclosure vs gonuts) — zero code, feeds deep-dive item 7; if it surfaces a live vulnerability, that gets its own P0 rather than blocking on this issue's say-so.
Post-0.6.0 deep dive — scope
- Enumerate every crash window (swap/mint/melt × before-POST/after-POST-before-save, plus send/receive hand-offs) — extend #497's research doc into the canonical table; re-pin the exact upstream saga sources (commit + paths) into #497's research doc so citations stop drifting.
- Design decision with evidence: (a) bbolt
pending_opsintents inside gonuts-tollgate (counter range + secrets + rs persisted with the counter increment in one tx), vs (b) CDK behind the AF_UNIX sidecar, vs (c) full migration (#305/#277). Constraints: MIPS, flash wear (#505), multi-process access (#504), the 16-module replace-directure dance. - TollGate-layer transaction record (#502) with the explicit refund-vs-late-grant policy #258/#403 never got.
- Reconciliation daemon (#500) + recovery classes — owned vs handed-out clawback as separate verbs with policy (companion issue, filed alongside this one).
- Conformance/fault-injection matrix as the gate (#503, the #535 lane, cloud-lab kill-at-boundary lanes) — no saga-style change lands without a SIGKILL-at-every-boundary test reproducing the incident first.
- CDK parity checklist derived from the saga sources — the invariant list this repo must satisfy regardless of which implementation backs it.
- NUT-13 derivation audit (Class A attack surface): audit gonuts-tollgate's deterministic derivation (BIP32 path over
keyset_id_intvs HMAC over the full keyset ID) against the Conduition disclosure; verify residue-collision guards and how much NUT-09/restoreexposure we have. Outcome: a clean bill recorded in cashu-audit, or a gonuts issue + pin bump. - Upstream recovery-semantics NUT: contribute TollGate's incident taxonomy to cashu-audit's
interrupted-operation-recoverydraft (venue: cashubtc/nuts) — crash-point state machines, idempotency keys, restore obligations. - Port
used_by_operationproof-ownership markers into the #500/#702 design — CDK's swap-saga tests assert them; they are the coordination primitive between the sweeper, in-flight reseller hand-offs, and clawback.
Acceptance criteria (deep dive, post-release)
- The crash-window table exists, is exhaustive, and each row cites a failing test that proves the window (red) before the fix (green).
- A written decision on (a)/(b)/(c) with measured trade-offs, checked into
docs/decisions/per ADR practice. - The #535 lane runs the full brick-family scenario set (10002-retry, restart-alias, timeout-retry, kill-between-increment-and-swap, kill-between-swap-and-save) against the chosen design and measures zero re-exposures.
Out of scope
Release management itself (#665, #339); individual P0 execution (tracked in their own issues); NUT-07 policy work (companion recovery-semantics issue).
- Dominant language
- Go
- Stars
- 12
- Forks
- 14
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 211
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from OpenTollGate/tollgate-module-basic-go
-
go-battery needs an ndsctl on PATH: TestPurchaseSessionGuardHoldsThroughTheOutcomeUnknownWindow fails on bare hosts (passes with stub)Possibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
OpenTollGate/tollgate-module-basic-go#726 ·
Maintainers usually reply within 1 day
-
rebrand-literal-gutter: uhttpd section-vocabulary check trips on a COMMENT (uhttpd.luci in 92-tollgate-admin-setup:178)Possibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
OpenTollGate/tollgate-module-basic-go#723 ·
Maintainers usually reply within 1 day
-
Discovery endpoint serves text/plain content-type on / — r2r clients warnPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
OpenTollGate/tollgate-module-basic-go#628 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
OpenTollGate/tollgate-module-basic-go#725 ·
Maintainers usually reply within 1 day
-
cloud-lab Dockerfile.client: mid-file ARG invisible to FROM — client and killer images unbuildable on docker/buildkit 29 (golang:-bookworm)Possibly taken @Amperstrand claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 25/100
OpenTollGate/tollgate-module-basic-go#724 ·
Maintainers usually reply within 1 day
All issues in OpenTollGate/tollgate-module-basic-go
Similar issues
-
kind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 4 days
-
bug needs-acceptance
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
vllm-project/semantic-router#4744 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
jaegertracing/jaeger#9794 ·
Maintainers usually reply within 1 day
-
ScalingModifiers formula fails with "formula returned non-float result" when expression evaluates to an integerPossibly taken @Sarthak-Pandey claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
kedacore/keda#8270 · 1 comment ·
Maintainers usually reply within 1 day