Windows flake: daemon-foreground-relaunch e2e cleanup hits ENOTEMPTY while the unawaited stop CLI still writes the state dir

Open Beginner friendly
  • Is something wrong?
#2,918 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
github-actions, javascript, node.js
Domain
cli, testing

Research direction

Start with test/core/daemon-foreground-relaunch.test.js around line 282 and compare cleanup retries in test/core/daemon.test.js and test/core/daemon-backfill-sweep-wiring.test.js. Keep and await the Windows stop child before gateway assertions and cleanup, and ensure it is stopped if cleanup runs first. Done when the test passes repeatedly on Windows and test-windows is green.

Written by the indexing model from the issue text.

Description

Symptom

On test-windows, test/core/daemon-foreground-relaunch.test.js:282 ("an unsupervised daemon run applies a staged central config and keeps the gateway answering") sometimes fails after every assertion has passed. The cleanup fs.rm(root, { recursive: true, force: true }) in the finally block throws:

[Error: ENOTEMPTY: directory not empty, rmdir 'C:\Users\RUNNER~1\AppData\Local\Temp\hyp-t-2puvEe\hyp-relaunch-e2e-eWyA2J\.hyp\hypaware'] { errno: -4051, code: 'ENOTEMPTY', syscall: 'rmdir' }

There were three sightings on 2026-11-06. Each one passed when the failed job was re-run once:

PR Run Head Directory that would not empty
#2913 42136275812 715c1cbb .hyp\hypaware
#2914 42139698957 attempt 1 3ac79c02 .hyp\hypaware\dev-telemetry
#2916 42142432583 attempt 1 20dad66b .hyp\hypaware

None of these PRs touched the daemon or this test. Under the test-windows landing gate, every sighting costs a re-run before a PR can land.

Likely cause

On win32 the test stops the daemon through the LLP 0300 file control channel. It spawns hyp daemon stop and does not keep a handle on that process:

if (process.platform === 'win32') {
  spawn(process.execPath, [path.join(REPO_ROOT, 'bin', 'hypaware.js'), 'daemon', 'stop'], { env, stdio: 'ignore' })
}

The test then waits only for the daemon to exit before fs.rm runs. The stop CLI is a second process with the same HYP_HOME and HYP_DEV_TELEMETRY=1. requestDaemonStop (src/core/daemon/runtime.js) polls until the daemon's pid is gone. Only after that does the CLI:

  • reconcile the pid files under the state root;
  • print daemon: stopped;
  • flush its own dev telemetry into <HYP_HOME>/hypaware/dev-telemetry (src/core/observability/logger.js, meter.js, tracer.js);
  • exit. That work overlaps the recursive delete:
  • Windows rmdir fails with ENOTEMPTY when an entry is created while the delete is running;
  • it also fails when a file is still open by a live process.

The #2914 sighting failing on dev-telemetry itself supports this reading. POSIX never hits the race because it stops the daemon with proc.kill('SIGTERM') in-process.

Proposed fix (test-only, bounded)

  1. Keep the stop child and await its exit before the gateway assertions and the cleanup. If it is still alive when finally runs, kill it.
  2. Give the cleanup the bounded retry the sibling daemon tests already use: fs.rm(root, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }). Examples are test/core/daemon.test.js:956 and test/core/daemon-backfill-sweep-wiring.test.js:139. That bounds the wait at about 1s worst case. Step 2 alone would hide the race rather than close it, so step 1 is the fix and step 2 is the backstop.

No product code changes. The stop CLI tidying pid files and flushing its telemetry after the daemon exits is the correct order.

Validation

Windows is gated by LLP 0584 #validation, and test-windows is a landing gate. The fix is proven by a green test-windows on the fixing head, plus the test passing in a loop on a Windows runner. Linux and macOS are unaffected because they take the SIGTERM path.

Logs from the three runs are held in the HypForge review records for #2913, #2914 and #2916.

Dominant language
JavaScript
Stars
11
Forks
0
Avg merge
2h 15m
Merged PRs (30d)
322

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from hyparam/hypaware

All issues in hyparam/hypaware

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.