Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Design box health tests for manufacturing QA and firmware releases

Aberta
  • Algo errado?
#339 0 comentários 0 reações 0 responsáveis Ver no GitHub

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
8/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
lua, shell
Domínio
testing-qa

Direção de pesquisa

This is a design proposal, not an implementation task: it asks for agreement on hardware profiles, the report contract and the verdict model. The payload names src/services/ui/diagnostics.lua, src/services/ui/http/request.lua (the /api/diagnostics stub) and the grand-pipeline runner-scripts/image_installer/execute_diagnostics.sh. Done means the review questions are answered and split into focused implementation issues; the first concrete step is making those consumers fail closed on stubs and count mismatches.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

enhancement

Goal

We need a repeatable way to establish that a newly assembled BigBox or GetBox works as a normal Wi-Fi box, qualify a firmware release on known-good hardware, and assess health after an update. These three workflows should share diagnostic checks and evidence, while differing in physical steps and the confidence they can claim.

Hardware acceptance must expose faults in required components, slots, ports, radios, power features, and client paths. Release qualification must test the same functional contract plus updates, persistence, recovery, and longer operation. An unattended post-update check should report what returned and what remains untested; without an external client probe, it cannot certify Wi-Fi client forwarding.

Why the old diagnostics are insufficient

The v0.10.7 diagnostics largely counted installed packages, running processes/services, modem entries, provisioned files, and basic cloud requests. Some checks prove less than their names suggest: modem enumeration returns an empty missing list even when its count is wrong, and the old UniFi check only verifies that an IP is configured.

The current Lua /api/diagnostics handler returns HTTP 200 with stub: true, zero expected checks, and empty failure lists. The BigBox diagnostic runner decides success solely from nonempty missing lists. Running that function against local fixtures returned “All diagnostic checks passed” for the stub, an expected-two/observed-one modem mismatch, {}, and malformed JSON. An explicit missing-package entry did fail. This path must cease being used as an acceptance gate until its consumers fail closed.

Boot-slot update success is also a different claim from functional health. A box can boot the intended image while a required component or client path remains broken.

Proposed testing model

Define a versioned expected hardware profile for each supported model and revision, independently of what the box discovers. Pair it with the exact release/image manifest. A missing required component must not cause the test suite to silently shrink. GetBox’s current default image runs legacy devicecode; the first design should include a legacy adapter, not assume every product has migrated to Lua.

Use one catalogue of checks with three profiles:

Profile Required evidence
Hardware acceptance Every required component and connector; each modem/SIM path; each required Wi-Fi band and client segment; wired and cellular backhauls; failover; fitted power/I/O responses; provisioning, persistence, reboot, and cloud operation
Release qualification Acceptance on known-good units, supported upgrade paths, host and component image identity, configuration persistence, recovery, shaping/policy detail, and soak
Post-update health Intended image and components running, service/config readiness, persistent state, and bounded probes over paths that were expected to work before update; explicitly record untested physical and client behavior

Test the client experience from an isolated external client: association on the intended radio, DHCP, DNS, forwarding, and a fresh response through the selected backhaul. Test each path separately; an aggregate “online” state can conceal a failed modem. SIM or cable removal remains an operator step, but the runner should verify the loss and recovery automatically. Use controlled, sustained client traffic to test shaping; the current short box-originated WAN speed estimate is not a shaping qualification test.

Every check should record its expected predicate, prerequisites, evidence source and acquisition time, path identity, bounded wait, and result. Results are PASS, FAIL, WARN, BLOCKED, N/A, or NOT_TESTED. An incomplete, stale, unsupported, partial, malformed, or stubbed result cannot become a pass. The overall verdict is FAIL for a failed required check, INCOMPLETE for unresolved required checks, and PASS only when the required suite is complete. Installation outcome and health verdict remain separate.

Start with a guided station runner and machine-readable report, reusing relevant capabilities from the existing QA tool. Keep a small independent on-box collector for post-update checks so failure of Lua or the local UI cannot erase the result. Share probes and report evidence with future field troubleshooting, while keeping factory and release acceptance criteria separate.

Proposed order of work

  1. Make existing diagnostic consumers reject stubs, bad HTTP/JSON, missing sections, and count mismatches. Show “unavailable/incomplete” in the UI instead of a health percentage.
  2. Agree the hardware profiles, connector mappings, required features, release provenance, station setup, and report contract. Pilot bigbox-v1-cm-2 and GetBox V1; add the other supported BigBox profiles before claiming complete product coverage.
  3. Build and validate the guided station workflow with known-good SIMs, backhaul, cables, an isolated client adapter, and a local probe server. Calibrate performance and recovery limits on representative good units.
  4. Add unattended post-update collection, correlation across reboot, deferred report delivery, and a release matrix covering supported upgrade and recovery paths. Retain existing boot/rollback policy until separately reviewed.

Review requested

  • Is the shared functional definition of a working box sufficient for hardware acceptance and release qualification?
  • Which model/revision and fitted-feature profiles should be in the first pilot?
  • Which checks should be required when a carrier or cloud dependency is unavailable? The proposal treats an unverified required check as incomplete rather than a hardware failure or a pass.
  • What equipment and acceptable per-box duration can the hardware team support?
  • What client canary, if any, can provide forwarding evidence after unattended updates?

Related: #329 tracks release functional differences; #306 concerns speed tests. This issue is for the acceptance design and its evidence contract, with implementation work to be split into focused issues after review.

Linguagem predominante
Lua
Estrelas
1
Forks
0
Merge médio
5d 15h
PRs com merge (30d)
3

Preparar o ambiente

Abrir no Codespaces

Inicia o contêiner de desenvolvimento do projeto no navegador, com a sua própria conta do GitHub.

  • Sem Dockerfile nem arquivo Docker Compose
  • Sem modelo de pull request
  • Sem guia de contribuição

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de jangala-dev/devicecode-lua

Todas as issues de jangala-dev/devicecode-lua

Issues semelhantes

Mais issues de Lua

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.