Harden smoke.sh and deployment helpers against RPC errors and reformatted JSON

Start in scripts/_deployment.sh: json_field and the alias lookup in assert_identities_match break on reformatted JSON. Reproduce with the How to verify snippet, using a copy of deployments/testnet.json under a temporary ROOT_DIR, and confirm compact and pretty-printed files give the same output. Then move to scripts/smoke.sh around line 40 (curl timeout) and lines 57-64 (the || true on reexecutor_info). Done when the acceptance checks pass with a fake stellar, and smoke.sh exits without calling register_reexecutor when that read fails.

Summary

scripts/smoke.sh and the shared scripts/_deployment.sh have several fragility problems. The smoke test can hang on a stuck health check, and a transient RPC error while reading the rogue re-executor's state makes it stake another 1,000 XLM on every run. It also never checks that the keystore still matches the deployment record, and the helper that reads deployments/*.json silently stops checking identities if the file is reformatted. No script checks that the tools it needs are installed.

Background

pnpm smoke (README, "Prove it works end to end") plays an agent, a challenger and a rogue re-executor against a local pnpm dev stack and real testnet. scripts/_deployment.sh is sourced by every script and resolves which deployment they act on; its assert_identities_match exists because the stellar-cli keystore is one flat namespace and another project can replace a key under the same alias (CONTRIBUTING, "Things that bite"). EscrowGate.register_reexecutor adds to existing stake rather than replacing it.

Current behaviour

1. scripts/smoke.sh, line 40: if ! curl -fs "http://127.0.0.1:$API_PORT/health/live" >/dev/null; then. curl has no --max-time, so a listener that accepts the connection and never answers hangs the script.
2. scripts/smoke.sh, lines 57-64:

  ROGUE_INFO="$(stellar contract invoke --id "$GATE_ID" --source "$ROGUE_ALIAS" --network "$NETWORK" \
    -- reexecutor_info --reexecutor "$ROGUE_ADDR" 2>/dev/null || true)"
  if ! echo "$ROGUE_INFO" | grep -q '"active": *true'; then
    echo "==> Staking '$ROGUE_ALIAS' (1,000 XLM)"

If the RPC call fails, || true turns that into an empty string, which is indistinguishable from "not staked", so the script stakes another 1,000 XLM. Repeated flaky runs drain the rogue identity.
3. scripts/smoke.sh never calls assert_identities_match (compare dev.sh, bootstrap_reexecutors.sh and approve_reexecutor.sh, which do), so a replaced deployer key is used to approve voters without warning.
4. scripts/_deployment.sh, line 44 and line 70:

json_field() { grep -o "\"$1\": *\"[^\"]*\"" "$DEPLOY_JSON" | cut -d'"' -f4; }
...
    recorded="$(grep -o "{\"alias\":\"$alias\",\"public_key\":\"[^\"]*\"" "$DEPLOY_JSON" | cut -d'"' -f8)"

json_field only matches "key": "value" with at most spaces after the colon; "key" : "value" or any reformatting returns an empty string. The alias lookup only matches the exact compact spelling {"alias":"rex1","public_key":"G..." that deploy_testnet.sh happens to write. If an editor or formatter pretty-prints deployments/<name>.json, recorded is empty and assert_identities_match skips the check without saying so (the [ -n "$recorded" ] guard).
5. No script checks for stellar, curl or openssl; a missing tool shows up as a cryptic command not found partway through.

Expected behaviour

• The health probe times out (a few seconds) and the script says the backend is not answering.
• A failing reexecutor_info call aborts the script with a FATAL: message and stakes nothing. Only a successful call that reports "not registered/not active" leads to staking.
• smoke.sh calls assert_identities_match right after require_deployment.
• json_field, deployment_is_public and the alias lookup in assert_identities_match return the same values for the committed compact file, for the same file pretty-printed with jq . or node -e 'console.log(JSON.stringify(JSON.parse(require("fs").readFileSync(0,"utf8")),null,2))', and for "key" : "value" spacing. If an alias in BOOTSTRAP_REEXECUTORS cannot be found in the record, assert_identities_match prints a WARN: line instead of silently skipping it.
• _deployment.sh provides a helper, for example require_commands cmd..., that lists every missing tool at once and exits non-zero; smoke.sh calls it for stellar curl node before doing anything else.

Proposed approach

1. Rewrite json_field and the alias lookup so they tolerate whitespace and line breaks. Either use node (already a hard prerequisite for everything these scripts start) to parse the JSON, or keep grep/sed but flatten with tr -d '\n' and allow [[:space:]]* around : and ,. Keep the function names and output unchanged, because every script calls them.
2. Add the WARN: for an alias that has no recorded key. Do not turn it into a failure; older records may not list every alias.
3. Add require_commands to _deployment.sh (it is sourced, so do not exit from a subshell; mirror the existing require_deployment style).
4. In smoke.sh: add require_commands, call assert_identities_match, add --max-time 5 (and --connect-timeout 2) to the curl, and replace the || true on the rogue read with an explicit exit-status check. Take the "no || true" fix only for the read of reexecutor_info; leave the other || true calls (funding an already-funded account) alone.
5. Confirm on your own deployment what reexecutor_info prints for an account that was never staked (null is expected) so that case still leads to staking on a first run.

Pitfall: deployment_is_public uses the same style of grep ("public": *true) and should get the same tolerance, since it protects public deployments from pnpm dev and pnpm smoke.

Files

• scripts/smoke.sh: preflight, identity check, curl timeout, no swallowed RPC errors.
• scripts/_deployment.sh: tolerant JSON helpers, missing-alias warning, require_commands.

Acceptance criteria

• Sourcing _deployment.sh with ROOT_DIR pointing at a temporary tree whose deployments/x.json is (a) the compact committed file, (b) the same file pretty-printed, (c) with "key" : "value" spacing, gives identical output from json_field escrow_gate_contract_id, and deployment_is_public is correct in all three.
• assert_identities_match detects a mismatched rexN key in the pretty-printed file (use a fake stellar returning a different address) and warns when an alias is absent from the record.
• With a fake stellar whose reexecutor_info call exits non-zero, smoke.sh exits non-zero without calling register_reexecutor.
• A listener that never responds makes the health probe fail within about 5 seconds.
• Running smoke.sh with curl hidden from PATH prints a message naming the missing tool.
• bash -n scripts/smoke.sh scripts/_deployment.sh passes (and shellcheck on smoke.sh, which sources _deployment.sh, if available).

How to verify

All checks except the last can be done without network or funds, using a temporary ROOT_DIR and a fake stellar script first on PATH:

tmp=$(mktemp -d); mkdir "$tmp/deployments"
cp deployments/testnet.json "$tmp/deployments/testnet.json"
ROOT_DIR="$tmp" bash -c 'source scripts/_deployment.sh; json_field escrow_gate_contract_id'
node -e 'const f=process.argv[1];require("fs").writeFileSync(f,JSON.stringify(JSON.parse(require("fs").readFileSync(f,"utf8")),null,2))' "$tmp/deployments/testnet.json"
ROOT_DIR="$tmp" bash -c 'source scripts/_deployment.sh; json_field escrow_gate_contract_id'   # same value

Then, with a local stack from pnpm dev and your own deployment (CONTRIBUTING, "Getting set up"), run pnpm smoke once and confirm it still passes.

Out of scope

• Changing the staking amount, the smoke scenarios, or backend/src/smoke/run.ts.
• Adding preflight checks to scripts other than smoke.sh (they can adopt require_commands later).
• Changing the deployment record format that deploy_testnet.sh writes.

Notes

Medium: a handful of independent shell fixes. If you prefer to keep the PR small, the JSON helper tolerance and the rogue-stake fix are the most valuable pieces, but the issue is complete only with all of them. Merge-conflict hotspot: scripts/_deployment.sh is also read by the sibling script issues, which do not edit it.