s3wal: promotion-readiness tracking — path from experimental to supported (beta → GA)

Promotion anchor for the s3wal zero-infrastructure backend, requested by the owner and produced from a full promotion-readiness assessment (read from origin/main @ c878adb5, the #133 epic record, and the committed campaign artifacts under docs/plans/s3wal-bench-data/). Epic #133 closed as completed (all five plan phases delivered); this issue is the sole anchor for the remaining path from experimental to supported.

Context: what "experimental" means today

The label is asserted consistently across every user-facing entry point:

• README.md:36 — "An experimental third backend collapses both stores into one"; the quickstart section is titled "zero-infrastructure mode, one bucket (experimental)".
• docs/reference/configuration.md:68 — "s3wal://bucket?endpoint=…&region=… (experimental zero-infra mode, single node…)".
• docs/how-to/deploy-production.md:436 — "## Zero-infrastructure mode (experimental)", plus the honest scaling-boundaries table (one node ever; one CAS point per repo group; S3-RTT latency floors; 4 MiB warn / 16 MiB refuse manifest cap; in-process jobs) and the lease framed as detection, not safety.
• docs/explanation/storage-engine.md:190, docs/reference/cli.md:16,48, docs/how-to/storage-lifecycle-and-sweeping.md:31, ADR-46 header — same label.
• Gate 1a-B verdicts on #133: "keep experimental — continue" (2026-08-27), with pre-committed flip condition on the starvation arm.

Gate structure (per docs/plans/s3wal-plan.md §3/§4 and docs/plans/s3wal-bench-results.md)

Gate	Status	Evidence
1a-A self-hosted correctness (SeaweedFS)	PASS	#142/#145; zero lost updates across 101k+ contention advances
1b conformance green	PASS	#156 — portable RefStore conformance, per-PR
2 DST ≥ nightly seed budget + crash matrix	PASS	#159 — W1–W9 matrix, real SIGKILL drill; dst-nightly backend: [sqlite, s3wal]
3 e2e lane + docs	PASS	#168, #175/#183 (all four Diátaxis quadrants)
4 bench publication + verdict	PASS	#171, #183 — six committed evidence artifacts
1a-B real-S3 kill thresholds	PARTIAL	#183: arms (1), (2), (4) + throughput/scaling pass; 30 s starvation arm recorded tripped-as-written (56.4 s @ N=8) under the pre-#184 runner, assessed as harness artifact, unresolved pending a conformant re-run — pre-committed flip condition: trips again under §2.3 backoff ⇒ recommend killing the mode

Remaining criteria

A. Performance gate completion (the plan's own recorded debt)

• A1 [BLOCKING] §2.3-conformant starvation re-run on real S3 (runner fixed in #184 after the campaign). The one open kill-criterion arm; nothing promotes while a kill arm is formally unresolved. (~40 min + AWS credentials.)
• A2 [BLOCKING] ≥ 3 separate-day headline repeats (plan §3 method); all real-S3 numbers are marked "preliminary single-day".
• A3 [BLOCKING] One campaign from in-region compute (recorded deviation: ~7 ms-RTT host near, not in, eu-central-1).
• A4 Post-#185 e2e re-measure + #182 census closure (CODESTORAGE_S3_OP_LOG seam landed for this). Non-blocking: the 2.8× uncontended-push gap is a deliberate, documented tradeoff — but the published 1,070 ms figure predates #185 and must be refreshed before any "supported" claim quotes numbers.

B. Correctness / durability

• Conformance, W1–W9 crash matrix, DST nightly, real-S3 require-mode qualification (22/22), zero lost updates — done.
• B1 [BLOCKING — delivered] The s3wal serve path runs in the per-PR test-s3wal CI job, including restart-from-bucket and dual-serve drills. The current workflow selects OPENSTORAGE_TEST_BACKEND=s3wal; this is no longer nightly-only. Verified current main CI passed all six jobs.

C. Operations

• C1 [BLOCKING — local procedure delivered] #248 delivered the stopped-node, same-binary whole-bucket backup/restore runbook and local SeaweedFS drill, including opaque keys and user metadata, missing/corrupt/metadata-mismatch rejection before boot, destruction of the owned source, authentication, Git/LFS recovery, and fresh-cache restart. All writers must be fenced; this does not bless a live aws s3 sync. Real-AWS/cross-region behavior, scale, RPO/RTO, and promotion remain unqualified and separately gated.
• C2 — partial: local cross-bucket mechanics covered. C1’s offline, same-binary procedure and historical SeaweedFS drill exercise recovery into a differently named empty bucket, with complete inventory verification and cold-cache acceptance checks. This is copy-and-cutover, not an in-place bucket rename. It does not establish general provider, cross-endpoint, or cross-region migration qualification. The broader criterion remains open. The maintainer desk owns target-specific contract/evidence selection when that qualification work is scheduled, subject to the existing cloud/access boundaries; no duplicate procedure or repeat drill is needed for the already-covered local mechanics.
• C3 [BLOCKING at GA — beta runbook delivered] #262 / PR #264 (merge fbd20684a03a95e1f5875142d86fbe470b6fb9e3) delivered source-backed operational signals, bounded interpretations, manifest-size guidance, and fenced shutdown/restart instructions. This is documentation verification, not incident or cloud qualification. Product-wide metrics and the GA observability criterion remain open. The bounded startup-diagnostic wording follow-up was delivered by #269 / PR #270 (merge 84101df0e27770f4808735fa41485bb7f94deaf6), preserving fail-closed lease behavior and adding no recovery mechanism.
• C4 Lease lifecycle — mature (TTL/cadence, store-clock expiry arms, SIGTERM release #176, dual-serve + restart drills passed on real S3).

D. Compatibility guarantees

• D1 [BLOCKING — delivered] #260 / PR #261 (merge ca803d0ac6ca90e3d0f62810c9aafc4b5b56c825) delivered the prospective v1 format policy anchored at d94125f63c0bec1ea4bdf833dd19deaaa22e4a9d, two operator navigation links, and metadata codec rejection tests. Both repository-manifest and metadata codecs reject unsupported version values. Optional fields are not automatically read/write-compatible; this is not historical release-pair qualification, a bucket-wide rollback guarantee, or an existing v2 migration. Post-merge CI passed all six jobs.
• D2 [BLOCKING at GA — local harness delivered] #267 / PR #268 (merge 6f8e8c0b8a452455d9c16fb7a66f00af5c8df20a) delivered the opt-in official-v0.15.0 → exact-candidate upgrade harness and local runbook. The official artifact → candidate 245bd6d2c7607acbe56236d0c583eb6155fe4161 passed the isolated SeaweedFS fixture with active leases, serial clean shutdown, cold-cache Git/LFS and selected metadata/authentication/encrypted-secret checks, candidate writes, and a second cold restart. This is exact-pair local evidence, not all-v1-data, AWS, or downgrade qualification. D2 remains open for final-published-pair execution, owned by the release maintainer after the release-publication hold is explicitly lifted; no future release is qualified here.

E. Functional parity vs the Postgres standard

• E1 [BLOCKING — decision, not necessarily fix] docs/explanation/wire-compatibility.md:43: a git-surface push does not auto-schedule sync/forwarding under SQLite or S3WAL (hook subprocesses do not use the serve process's in-process queue). API writes use the serve-side queue. Implementation is now owned by #266, with a test-first feasibility gate for authorized post-CAS intent and bounded live consumption using existing persisted semantics. Current event emission is after the ref CAS, not atomic with it; existing pending-work recovery is a one-shot boot callback, not a periodic reconciliation loop. Any implementation must preserve caller read/policy restrictions and loopback suppression. E1 remains open until implementation and its acceptance gates pass; the intended contract is best-effort post-commit scheduling, not atomic or at-least-once delivery.

F. Documentation

• All four quadrants shipped and review-hardened (#175, #183).
• F1 Runbooks falling out of C1/C2/C3/D1. The C1 stopped-node backup/restore runbook is delivered by #248, the D1 prospective format policy by #261, and the C3 beta operational runbook by #264. Migration and release-pair upgrade qualification remain separately tracked. Blocking status inherits from the corresponding criterion.

G. Support matrix

• Qualified today: SeaweedFS (CI reference) + AWS S3 (require-mode lane + campaign). MinIO explicitly not qualified (broken conditional-write semantics, documented). No GCS/other claims.
• G1 [BLOCKING — conditional policy delivered; promotion still gated] #263 / PR #265 (merge 0338139a2952532fb5dfbe89d864062d07025625) delivered the conditional AWS S3 + SeaweedFS target matrix and separated ordinary blob capabilities, historical qualification evidence, and S3WAL support status. All S3WAL targets remain experimental. The supported list at promotion still depends on the remaining gates and release-specific qualification; this documentation change is not new provider qualification.
• G2 [BLOCKING at GA] Standing real-S3 qualification lane (scheduled TestQualifyExternalEndpoint + bench cells; needs an infra decision on a long-lived test bucket/role). Nightly SeaweedFS suffices for beta.

Recommended promotion sequence

Stage 1 — experimental → beta ("supported for evaluation; format-stable"): A1 + A2 + A3, B1, D1, C1, G1. The 2.8× push cost ships as a documented boundary.

Stage 2 — beta → GA ("production-supported"): C3, D2, A4/#182 closure with refreshed published numbers, E1 decision executed, G2, plus one full multi-day bench series recorded post-#185.

Kill-path note (unchanged from the recorded verdict): if A1's conformant re-run trips the 30 s starvation line again, the pre-committed flip condition applies and this issue converts to a deprecation plan.

---

Progress updated 2026-09-14; original assessment retained above.

Generated with xum • Model: coder:openai/gpt-6-astra • Thinking: high