Decide whether components get a quality scale

Open
#206 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
github-actions

Research direction

Start with the proposed rules in this issue, then read AGENTS.md, the logging guide, and the referenced review changes to understand the existing reliability standards. Decide whether ESPHome should adopt a component quality scale or a single long-uptime marker, and settle the criteria, display location, ownership, and regression policy before implementation.

Written by the indexing model from the issue text.

Description

Summary

Should ESPHome have a quality scale for components, like Home Assistant's integration quality scale but built around our reliability and uptime rules? The core is now at the point where a device can expect years of uptime from a software point of view, but components vary a lot, and nothing tells users or contributors which ones meet that bar.

Where things stand

  • Most, if not all, of the heap churn and fragmentation in core is gone, so long-running devices no longer slowly cut up their heap.
  • Runtime crash reporting is in place, so the crashes that are left show up instead of looking like random reboots.
  • Crashes are way down, and most of the ones still reported come from upstream code (SDKs and libraries), not from ESPHome. They are still ours to fix, because we own the product end to end, but it is a lot better than it was.
  • From a software point of view, most devices can now expect at least 10 years of uptime.
  • Components have not all caught up. Some still allocate on the heap after setup(), and some have other long-uptime problems. Today there is no list of which ones, and no way for a user to tell before they pick one.

Why it might be worth it

  • Contributors: a visible scale with clear steps tends to get people improving components on their own, because they can see what "better" means and claim it.
  • Users: people choosing between two drivers for the same chip could pick the one that is known to be safe for long uptime.
  • Industrial use: that is not a goal of the project, but a clear statement of which components meet a long-uptime bar opens that door without us having to promise anything for the rest.

How Home Assistant does it

Home Assistant grades integrations Bronze, Silver, Gold and Platinum (quality scale). Each tier is a list of concrete rules, and an integration meets every rule of its tier and the tiers below. Each integration keeps a quality_scale.yaml that marks every rule done, todo or exempt (with a reason). Hassfest fails CI when a rule for the claimed tier is neither done nor exempt, reviewers check the rules a script cannot, and the tier is shown on the docs page and in the UI.

Their tiers are about setup flows, translations, discovery and typing. Ours would be about what keeps a device running for years, taken from the rules in AGENTS.md and the logging guide, plus the cleanups reviews keep missing that esphome/esphome#22708 adds to AGENTS.md and the review skill.

A possible shape (starting point only)

  • Bronze, builds and behaves:
    • schema validated, and script/build_language_schema.py extracts it
    • build tests on every platform the component supports
    • dump_config() reports the setup; failures go through mark_failed() with a reason, in setup() only
    • logging follows the logging guide: readings at VERBOSE, no "Setting up" lines, related dump_config() lines in one call
    • a code owner
  • Silver, safe to run for years:
    • no heap allocation after setup() (fixed-size containers, StaticVector, StringRef)
    • scheduler lambdas capture at most 8 bytes; repeating work uses set_interval, never a self-rearming set_timeout
    • no millis() in loop(), and no rate-limit gate shorter than the 16 ms loop
    • runtime read and write failures set status_set_warning() and the next success clears it
    • multi-step command sequences never advance on a malformed answer
    • no state that copies other state (a flag that mirrors a timer, a has_x_ bool beside x_)
  • Gold, lean and proven:
    • constants and string tables kept in flash on ESP8266 (PROGMEM, LOG_STR, PROGMEM_STRING_TABLE); one table where a value is both encoded and decoded
    • members laid out without padding, small codes as uint8_t, flags packed
    • math that depends only on the config done in to_code, and model-specific code behind a cg.add_define()
    • leaf classes final, no dead guards or unused helpers
    • C++ unit tests or integration tests that fail if any path regresses, not only build tests
    • tested on real hardware, with the board and result recorded
  • Platinum (maybe), long-run verified:
    • a soak test with no heap growth and no crash reports over a set period
    • no exemptions

Enforcement ideas

  • A per-component rules file, like Home Assistant's quality_scale.yaml, with the tier claimed in the component's __init__.py; CI checks that every rule for the tier is done or exempt.
  • Some rules can be checked automatically. For example, the host integration tests could wrap malloc and fail when a component allocates after setup().
  • The rest stay with reviewers, as in Home Assistant. The review skill from esphome/esphome#22708 could label each finding with its tier, so an automated review also tells a contributor what the component needs for the next one.

To discuss

  • Do we want a full scale, or only a single "long-uptime ready" marker (roughly Silver above)?
  • Are these the right rules for each tier, and is the order right?
  • Where is the tier shown: the docs page, device builder, or both?
  • Who assigns a tier, and what happens when a component regresses?
Dominant language
No language data
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from esphome/backlog

All issues in esphome/backlog

Similar issues

More Backend & API Design issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.