Flaky: Linux owlet sometimes fails on power-loss-truncated corpus hoots (owlet-failed vs incomplete-read)

Open
  • Is something wrong?
#4 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
backend, testing

Research direction

Start with readers/hoot.py::_run and the named corpus and backup tests, then inspect how quarantine reasons and owlet output are recorded. Run the Linux owlet repeatedly on the two named truncated hoot files and compare exit status, stderr, and wpilog output. Done when the intermittent failure is characterized and the tests verify a reliable handling strategy for it.

Written by the indexing model from the issue text.

Description

bug

Seen on PR IgniteRobotics/flashpoint-private#15, CI run 37319717473, corpus job attempt 1. The same commit passed on rerun, and passed on two earlier runs.

The GACMP_E10 truncated hoots (GACMP_E10_rio_2026-04-11_19-26-19.hoot, GACMP_E10_6E94…_2026-04-11_19-29-15.hoot) were quarantined as owlet-failed instead of being ingested with incomplete-read. This failed:

  • tests/test_ingest_corpus.py::test_expected_successes_and_quarantines
  • test_truncated_hoot_flagged_short_coverage
  • tests/acquire/test_backup_corpus.py::test_restore_drill_reproduces_the_ledger_and_gold. In this one the original lake converted the hoot and the rebuild didn't, all within the same run.

readers/hoot.py::_run treats a non-zero owlet exit as owlet-failed unless the incomplete-read marker appears in stdout or stderr. The Linux owlet appears to be nondeterministic on truncated hoots: it sometimes exits non-zero without the marker. The macOS owlet exits 0 on this file in 20 of 20 runs, so this can't be reproduced locally. The owlet stderr detail isn't in the CI log.

Next steps:

  • Log the owlet exit code and the last stderr line in the quarantine reason detail.
  • Loop the Linux owlet on these files to characterise the failure.
  • Consider a single retry, or accepting a non-zero exit when a non-empty wpilog was written.

🤖 Generated with Claude Code

Dominant language
Python
Stars
0
Forks
1
Avg merge
18h 37m
Merged PRs (30d)
15

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from IgniteRobotics/flashpoint

All issues in IgniteRobotics/flashpoint

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.