crash-window lane: no value recovery after restart (SIGKILL between mint-signed and wallet-save) — #502/#700 acceptance FAILS on main

Aperta
  • Qualcosa non va?
#719 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

  • #771 di @Amperstrand — chiusa senza merge
  • #873 di @Amperstrand — chiusa senza merge

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
docker, go, python
Ambito
backend, databases, testing

Direzione di ricerca

Start with tests/cloud-lab/run-crash-injection.sh and identify the post-restart recovery wait and assertion. Then follow the receive-intent journal and resume in src/merchant/ and the swap-intent machinery in src/tollwallet/; compare the lane behavior with the passing unit test named in the issue. Determine whether reconciliation completes within the lane window and whether it recovers value; done means the lane exits 0 and a fresh payment succeeds.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

The crash-window acceptance lane for the payment-convergence work (#502/#700, the business-transaction record) fails on current main: after the tollgate is SIGKILLed between "mint signed the swap" and "wallet has persisted", the boot-time intent resume does not recover the value within the lane's window. This is the exact invariant the lane exists to hold.

Severity: S1/S2 candidate (funds-adjacent). The log shows the #502 machinery engaging (intent restored, NUT-07 reconciliation started) — so the failure is either reconciliation-not-completing in the lane's wait window (timing) or reconciliation completing without recovering the value (lost value). Distinguishing those two is the first triage step.

Verified on

  • Commit: 68ad5144ab09cb4723d8f3f655cbba1ac7ea9de5 (main, 2026-10-07) + release-prep metadata commit (code-identical)
  • Host: docker 29.1.3, linux/amd64; lane run via tests/cloud-lab/run-crash-injection.sh
  • Campaign evidence: release-labgrid-testing, 2026-10-07

Evidence (verbatim, from the lane transcript)

== paying in the background (the swap response will be swallowed)
== waiting for the swap response marker (mint has signed)
   marker: /v1/swap
== SIGKILL the tollgate (mid-wait: mint signed, wallet has NOT processed)
   killed at 18:53:33
== restarting the tollgate; boot resume must replay the intent
FAIL: no recovery after restart — the crash window destroyed value:
2026/10/07 16:53:33 PurchaseSession: Receive completed, amount=0, err=mint did not answer; the outcome is unknown: could not swap proofs: mint http://killer:8085/v1/swap did not answer; the request may have been processed — reconcile before retrying: Post "http://killer:8085/v1/swap": EOF
2026/10/07 16:53:34 Restored 1 pending receive intent(s) — reconciling against their mints (NUT-07)

Note the two log lines: the pending intent IS restored and NUT-07 reconciliation IS started one second after restart — then the lane's recovery assertion fails.

Reproduce

Prerequisites: a linux docker host (docker 29.x needs the Dockerfile.client ARG fix from the companion issue "cloud-lab client image unbuildable on docker/buildkit 29" — or build the client image manually first). If your docker build bridge has no working DNS for proxy.golang.org (symptom: dial tcp: lookup proxy.golang.org ... network is unreachable inside RUN go build), pre-build images with build.network: host.

git clone https://github.com/OpenTollGate/tollgate-module-basic-go && cd tollgate-module-basic-go
git checkout 68ad5144ab09cb4723d8f3f655cbba1ac7ea9de5
cd tests/cloud-lab
export TG_GO_VERSION="$(jq -r '.go.version' ../../packaging/build-inputs.json)"   # 1.26.8
./run-crash-injection.sh ; echo EXIT=$?

Expected (per #497/#502/#676 acceptance): exit 0 — intent resume recovers the value AND a fresh payment afterwards still succeeds.
Actual: FAIL: no recovery after restart — the crash window destroyed value and exit 1.

Troubleshooting guide (for the fixing agent)

  1. Read the lane first: tests/cloud-lab/run-crash-injection.sh — the killer proxy (killerproxy.py from PRTA, mounted read-only) swallows every /v1/swap response once the mint has signed; the tollgate is SIGKILLed when the /marker/kill.marker file appears; the assertion block is after the restart. Find the exact wait window it gives the reconcile loop.
  2. Distinguish timing vs lost value: instrument or watch the upstream container logs after restart — does the NUT-07 reconcile (Restored 1 pending receive intent(s)) ever complete? If it completes but the proofs are gone, that's lost value; if it needs longer than the lane waits, the lane's window is too tight for the reconcile path (still a release problem, different fix).
  3. Code pointers: the receive-intent journal and resume live in the #502/#700 business-transaction work — src/merchant/ (purchase-session guard + journal; see purchasesession_duplicate_guard_test.go for the intended semantics) and src/tollwallet/ (swap-intent machinery; gonuts-tollgate v0.13.0 per src/tollwallet/go.mod). The "outcome is unknown … reconcile before retrying" string in the log above is the unknown-outcome path — that is the state the resume must resolve.
  4. Reference material: docs/architecture/bearer-instrument-port.md and research/bearer-port-demo/ describe the intended derivation-replay recovery (NUT-07 answer is about consumption, never existence; SPENT/UNSPNT only from what the authority said).
  5. Unit-level repro that passes today: cd src/merchant && go test -tags testenv -run 'TestPurchaseSessionGuardHoldsThroughTheOutcomeUnknownWindow' -v (needs an ndsctl shim on PATH on a bare host — see companion issue). The docker lane is where it fails; the unit test seeds a scripted wallet, the lane uses a real mint behind the killer.

Fix hints

  • If timing: widen/loop the lane's post-restart wait on a functional signal (e.g. poll the owed-grant/session event rather than a fixed sleep), and check the reconcile backoff constants the boot resume uses.
  • If lost value: the boot-time replay must re-derive the blinded proofs from the journal and re-drive the swap (or recover via NUT-09 token state) — start where Restored N pending receive intent(s) is logged and follow what that reconcile actually does for a swap whose response was swallowed.
  • Regression guard: this lane IS the regression guard — once fixed, keep it in the release gate (make release-check with TOLLGATE_RELEASE_CHECK_CONFORMANCE=1 on the machine that owns the lane).

References

  • #497 (crash-window acceptance), #502/#700 (business-transaction record), #676 (harness), #631 (bearer-instrument port research)
Lingua principale
Go
Stelle
12
Fork
14
Merge medio
3g 20h
PR unite (30g)
261

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di OpenTollGate/tollgate-module-basic-go

Tutte le issue di OpenTollGate/tollgate-module-basic-go

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.