Design box health tests for manufacturing QA and firmware releases

This is a design proposal, not an implementation task: it asks for agreement on hardware profiles, the report contract and the verdict model. The payload names src/services/ui/diagnostics.lua, src/services/ui/http/request.lua (the /api/diagnostics stub) and the grand-pipeline runner-scripts/image_installer/execute_diagnostics.sh. Done means the review questions are answered and split into focused implementation issues; the first concrete step is making those consumers fail closed on stubs and count mismatches.

Goal

We need a repeatable way to establish that a newly assembled BigBox or GetBox works as a normal Wi-Fi box, qualify a firmware release on known-good hardware, and assess health after an update. These three workflows should share diagnostic checks and evidence, while differing in physical steps and the confidence they can claim.

Hardware acceptance must expose faults in required components, slots, ports, radios, power features, and client paths. Release qualification must test the same functional contract plus updates, persistence, recovery, and longer operation. An unattended post-update check should report what returned and what remains untested; without an external client probe, it cannot certify Wi-Fi client forwarding.

Why the old diagnostics are insufficient

The v0.10.7 diagnostics largely counted installed packages, running processes/services, modem entries, provisioned files, and basic cloud requests. Some checks prove less than their names suggest: modem enumeration returns an empty missing list even when its count is wrong, and the old UniFi check only verifies that an IP is configured.

The current Lua /api/diagnostics handler returns HTTP 200 with stub: true, zero expected checks, and empty failure lists. The BigBox diagnostic runner decides success solely from nonempty missing lists. Running that function against local fixtures returned “All diagnostic checks passed” for the stub, an expected-two/observed-one modem mismatch, {}, and malformed JSON. An explicit missing-package entry did fail. This path must cease being used as an acceptance gate until its consumers fail closed.

Boot-slot update success is also a different claim from functional health. A box can boot the intended image while a required component or client path remains broken.

Proposed testing model

Define a versioned expected hardware profile for each supported model and revision, independently of what the box discovers. Pair it with the exact release/image manifest. A missing required component must not cause the test suite to silently shrink. GetBox’s current default image runs legacy devicecode; the first design should include a legacy adapter, not assume every product has migrated to Lua.

Use one catalogue of checks with three profiles:

Test the client experience from an isolated external client: association on the intended radio, DHCP, DNS, forwarding, and a fresh response through the selected backhaul. Test each path separately; an aggregate “online” state can conceal a failed modem. SIM or cable removal remains an operator step, but the runner should verify the loss and recovery automatically. Use controlled, sustained client traffic to test shaping; the current short box-originated WAN speed estimate is not a shaping qualification test.

Every check should record its expected predicate, prerequisites, evidence source and acquisition time, path identity, bounded wait, and result. Results are PASS, FAIL, WARN, BLOCKED, N/A, or NOT_TESTED. An incomplete, stale, unsupported, partial, malformed, or stubbed result cannot become a pass. The overall verdict is FAIL for a failed required check, INCOMPLETE for unresolved required checks, and PASS only when the required suite is complete. Installation outcome and health verdict remain separate.

Start with a guided station runner and machine-readable report, reusing relevant capabilities from the existing QA tool. Keep a small independent on-box collector for post-update checks so failure of Lua or the local UI cannot erase the result. Share probes and report evidence with future field troubleshooting, while keeping factory and release acceptance criteria separate.

Proposed order of work

1. Make existing diagnostic consumers reject stubs, bad HTTP/JSON, missing sections, and count mismatches. Show “unavailable/incomplete” in the UI instead of a health percentage.
2. Agree the hardware profiles, connector mappings, required features, release provenance, station setup, and report contract. Pilot bigbox-v1-cm-2 and GetBox V1; add the other supported BigBox profiles before claiming complete product coverage.
3. Build and validate the guided station workflow with known-good SIMs, backhaul, cables, an isolated client adapter, and a local probe server. Calibrate performance and recovery limits on representative good units.
4. Add unattended post-update collection, correlation across reboot, deferred report delivery, and a release matrix covering supported upgrade and recovery paths. Retain existing boot/rollback policy until separately reviewed.

Review requested

• Is the shared functional definition of a working box sufficient for hardware acceptance and release qualification?
• Which model/revision and fitted-feature profiles should be in the first pilot?
• Which checks should be required when a carrier or cloud dependency is unavailable? The proposal treats an unverified required check as incomplete rather than a hardware failure or a pass.
• What equipment and acceptable per-box duration can the hardware team support?
• What client canary, if any, can provide forwarding evidence after unattended updates?

Related: #329 tracks release functional differences; #306 concerns speed tests. This issue is for the acceptance design and its evidence contract, with implementation work to be split into focused issues after review.