Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Design box health tests for manufacturing QA and firmware releases

Aperta
  • Qualcosa non va?
#339 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
8/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
lua, shell
Ambito
testing-qa

Direzione di ricerca

This is a design proposal, not an implementation task: it asks for agreement on hardware profiles, the report contract and the verdict model. The payload names src/services/ui/diagnostics.lua, src/services/ui/http/request.lua (the /api/diagnostics stub) and the grand-pipeline runner-scripts/image_installer/execute_diagnostics.sh. Done means the review questions are answered and split into focused implementation issues; the first concrete step is making those consumers fail closed on stubs and count mismatches.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Goal

We need a repeatable way to establish that a newly assembled BigBox or GetBox works as a normal Wi-Fi box, qualify a firmware release on known-good hardware, and assess health after an update. These three workflows should share diagnostic checks and evidence, while differing in physical steps and the confidence they can claim.

Hardware acceptance must expose faults in required components, slots, ports, radios, power features, and client paths. Release qualification must test the same functional contract plus updates, persistence, recovery, and longer operation. An unattended post-update check should report what returned and what remains untested; without an external client probe, it cannot certify Wi-Fi client forwarding.

Why the old diagnostics are insufficient

The v0.10.7 diagnostics largely counted installed packages, running processes/services, modem entries, provisioned files, and basic cloud requests. Some checks prove less than their names suggest: modem enumeration returns an empty missing list even when its count is wrong, and the old UniFi check only verifies that an IP is configured.

The current Lua /api/diagnostics handler returns HTTP 200 with stub: true, zero expected checks, and empty failure lists. The BigBox diagnostic runner decides success solely from nonempty missing lists. Running that function against local fixtures returned “All diagnostic checks passed” for the stub, an expected-two/observed-one modem mismatch, {}, and malformed JSON. An explicit missing-package entry did fail. This path must cease being used as an acceptance gate until its consumers fail closed.

Boot-slot update success is also a different claim from functional health. A box can boot the intended image while a required component or client path remains broken.

Proposed testing model

Define a versioned expected hardware profile for each supported model and revision, independently of what the box discovers. Pair it with the exact release/image manifest. A missing required component must not cause the test suite to silently shrink. GetBox’s current default image runs legacy devicecode; the first design should include a legacy adapter, not assume every product has migrated to Lua.

Use one catalogue of checks with three profiles:

Profile Required evidence
Hardware acceptance Every required component and connector; each modem/SIM path; each required Wi-Fi band and client segment; wired and cellular backhauls; failover; fitted power/I/O responses; provisioning, persistence, reboot, and cloud operation
Release qualification Acceptance on known-good units, supported upgrade paths, host and component image identity, configuration persistence, recovery, shaping/policy detail, and soak
Post-update health Intended image and components running, service/config readiness, persistent state, and bounded probes over paths that were expected to work before update; explicitly record untested physical and client behavior

Test the client experience from an isolated external client: association on the intended radio, DHCP, DNS, forwarding, and a fresh response through the selected backhaul. Test each path separately; an aggregate “online” state can conceal a failed modem. SIM or cable removal remains an operator step, but the runner should verify the loss and recovery automatically. Use controlled, sustained client traffic to test shaping; the current short box-originated WAN speed estimate is not a shaping qualification test.

Every check should record its expected predicate, prerequisites, evidence source and acquisition time, path identity, bounded wait, and result. Results are PASS, FAIL, WARN, BLOCKED, N/A, or NOT_TESTED. An incomplete, stale, unsupported, partial, malformed, or stubbed result cannot become a pass. The overall verdict is FAIL for a failed required check, INCOMPLETE for unresolved required checks, and PASS only when the required suite is complete. Installation outcome and health verdict remain separate.

Start with a guided station runner and machine-readable report, reusing relevant capabilities from the existing QA tool. Keep a small independent on-box collector for post-update checks so failure of Lua or the local UI cannot erase the result. Share probes and report evidence with future field troubleshooting, while keeping factory and release acceptance criteria separate.

Proposed order of work

  1. Make existing diagnostic consumers reject stubs, bad HTTP/JSON, missing sections, and count mismatches. Show “unavailable/incomplete” in the UI instead of a health percentage.
  2. Agree the hardware profiles, connector mappings, required features, release provenance, station setup, and report contract. Pilot bigbox-v1-cm-2 and GetBox V1; add the other supported BigBox profiles before claiming complete product coverage.
  3. Build and validate the guided station workflow with known-good SIMs, backhaul, cables, an isolated client adapter, and a local probe server. Calibrate performance and recovery limits on representative good units.
  4. Add unattended post-update collection, correlation across reboot, deferred report delivery, and a release matrix covering supported upgrade and recovery paths. Retain existing boot/rollback policy until separately reviewed.

Review requested

  • Is the shared functional definition of a working box sufficient for hardware acceptance and release qualification?
  • Which model/revision and fitted-feature profiles should be in the first pilot?
  • Which checks should be required when a carrier or cloud dependency is unavailable? The proposal treats an unverified required check as incomplete rather than a hardware failure or a pass.
  • What equipment and acceptable per-box duration can the hardware team support?
  • What client canary, if any, can provide forwarding evidence after unattended updates?

Related: #329 tracks release functional differences; #306 concerns speed tests. This issue is for the acceptance design and its evidence contract, with implementation work to be split into focused issues after review.

Lingua principale
Lua
Stelle
1
Fork
0
Merge medio
5g 15h
PR unite (30g)
3

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Nessun Dockerfile né file Docker Compose
  • Nessun modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di jangala-dev/devicecode-lua

Tutte le issue di jangala-dev/devicecode-lua

Issue simili

Altre issue su Lua

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.