QCFilterTransform emits on every out-of-range record; emit on transitions instead

Open
#717 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
54/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
python

Research direction

Start in logger/transforms/qc_filter_transform.py, reading transform() around lines 60–110, then compare the stateful transition handling in GeofenceTransform at geofence_transform.py:258–261. The key design point is the default compatibility behavior, so read server/logger_manager.py:369–389 and the set_active_mode implementations before coding. Done means per-field in/out-of-bounds state emits once on entering a bad state, an optional resume_message fires when all fields recover, and steady bad values no longer repeat the command.

Written by the indexing model from the issue text.

Description

QCFilterTransform returns its message for every record that has a value out of bounds, not when a value goes out of bounds. It keeps no state between calls (logger/transforms/qc_filter_transform.py, transform() lines 60–110): each record is checked against bounds, and if any field fails, the message (or the joined error list) is returned. A sensor that sits at a bad value for an hour therefore produces the message at the sensor's full record rate.

Why it matters

On its own that is noise in whatever the message goes to (a logfile, an EmailWriter, Sealog). But the pattern the v2.7.0 documentation shows for conditional control makes it harmful: a QC logger whose writer is a LoggerManagerWriter, with message set to a logger manager command such as set_active_mode safe. Every implementation of set_active_mode (server/sqlite_server_api.py:441, server/in_memory_server_api.py:165, and the Django and FastAPI equivalents) overwrites every logger's active configuration with the mode's and signals an update.

The logger manager is careful about this: _apply_configs (server/logger_manager.py:369–389) restarts only loggers whose configuration actually changed, and _journal_mode records a mode change only when the mode differs. So a repeated command is harmless to loggers already in the mode's configuration. It is not harmless to a logger an operator has switched by hand in the meantime: that logger's active configuration is different from the mode's, so on the very next out-of-range record it is switched back, restarted, and journaled as config_changed, and that repeats at the sensor's record rate for as long as the value stays out of range. The operator's change cannot stick.

GeofenceTransform already does this the right way: it remembers last_position_inside and emits only when the inside/outside state changes (geofence_transform.py:258–261), with a seconds_between_checks throttle.

Proposal

Make QCFilterTransform emit on transitions, keeping per-field in/out-of-bounds state:

  • emit message (or the error list) when a field first goes out of bounds, and not again while it stays out;
  • add an optional resume_message emitted when all fields are back within bounds, mirroring TimeoutReader/TimeoutWriter (and GeofenceTransform's entering/leaving pair);
  • optionally a seconds_between_checks throttle as in GeofenceTransform.

Compatibility: anyone relying on the per-record behaviour (e.g. counting bad records downstream) could be given an emit_every_record: true flag that preserves the current behaviour, or the transition behaviour could be opt-in via on_transition_only: true for one release. The former seems the better default, since per-record emission is almost never what a logger-manager command wants.

With transition-only emission, a hand-switched logger is reset at most once per excursion, which matches what a genuine mode change does and is what an operator would expect.

🤖 Generated with Claude Code

Dominant language
Python
Stars
57
Forks
30
Avg merge
3h 14m
Merged PRs (30d)
56

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OceanDataTools/openrvdas

All issues in OceanDataTools/openrvdas

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.