[rush] rush-client-core: a failed or slow daemon startup leaves a durable .starting reservation that wedges the workspace (every command waits ~16 s, daemon start refuses); an invalid RUSH_* env value triggers it and hides the real error

The issue is in rush-client-core's daemon startup logic. Key files are libraries/rush-client-core/src/DaemonStartup.ts (lines 104-118) for reservation retention, libraries/rush-client-core/src/connectOrStartDaemon.ts (lines 236-243) for connection rejection, and libraries/rush-daemon/src/RushDaemonHost.ts (lines 274-299) for restart handling. Start by examining the .starting file lifecycle and the validation of environment variables before restart. The goal is to ensure failed startups release the reservation and errors are shown promptly.

Summary

runDaemonStartupAsync deliberately retains <lockfile>.starting when the daemon is not protocol-ready within the helper timeout, or when the launcher exits before readiness. Nothing ever removes it:

• tryConnectAsync treats any existing .starting file as "pending", so even a healthy, ready daemon is rejected;
• the starter loop spins until the 15 s deadline, then throws "unresolved startup handoff; refusing another launch", and every command pays about 16 s before falling back in-process;
• rush-client daemon stop does not clear it and daemon start refuses. There is no TTL and no owner check (the token holds no PID or start time).

Recovery requires manually deleting a file in $XDG_RUNTIME_DIR/rushd-<uid>/. We hit the wedge four independent ways: a daemon slow to become ready (SIGSTOP for 16 s as a stand-in for a heavy cold start), kill -9 of the daemon or of the startup helper during a cold start, and a single request with an invalid Rush environment value.

That last trigger is the worst. RUSH_ALLOW_WARNINGS_IN_SUCCESSFUL_BUILD=yes rush-client build makes the healthy warm daemon plan a restart (the request environment differs). RushDaemonHost closes the old daemon before the successor is proven viable. The successor throws in EnvironmentConfiguration.validate and exits 1, and the reservation is kept. The user only sees Daemon startup has an unresolved startup handoff at .../rushd-<hash>.pid.json.starting; refusing another launch, never the invalid-value message that native Rush prints in 0.8 s. Every later valid command then takes about 16.6 s.

Repro steps

RUSH_DAEMON=1 rush-client build; RUSH_DAEMON=1 rush-client build      # warm daemon
RUSH_DAEMON=1 RUSH_ALLOW_WARNINGS_IN_SUCCESSFUL_BUILD=yes rush-client build   # 16.5 s, exit 1, misleading message
RUSH_DAEMON=1 rush-client build                                        # 16.6 s every time, falls back in-process
rush-client daemon stop       # exit 1 "Could not connect"
rush-client daemon start      # 15.7 s, exit 1, same message

Expected result: An invalid environment value fails fast with the native error, and the daemon stays usable for other requests. A startup reservation is bounded and verifiable: it is released when the launcher exits without publishing an endpoint, treated as stale when its owner processes are dead, and ignored when the lockfile owner answers hello/ping. daemon stop/start can recover from a stale reservation.

Actual result: Permanent wedge with a 16 s penalty on every command, and the real cause is hidden. The launcher log contains Error: Invalid value "yes" for the environment variable RUSH_ALLOW_WARNINGS_IN_SUCCESSFUL_BUILD ... at WorkspaceSessionProvider.createAsync followed by DaemonClientError: Launcher exited (1) before protocol readiness; startup reservation retained.

Details

Root cause (main @ 60007c9a8c):

• libraries/rush-client-core/src/DaemonStartup.ts:104-118: the reservation is retained on timeout or early launcher exit (by design, per the rush-client-core README).
• libraries/rush-client-core/src/connectOrStartDaemon.ts:236-243: an existing .starting rejects the connection. :110-121 spins to the deadline. :70, :448: 15 s deadline.
• libraries/rush-daemon/src/RushDaemonHost.ts:274-299: #restartOnceAsync closes the healthy daemon before the successor is viable.
• apps/rush-cli-client/src/launchClient.ts:162: executeWithDaemonRestartAsync sits outside the fallback try/catch (:111-124), and result.errorMessage is never printed (:200-207).

Suggested fix: validate request-scoped Rush environment variables (the EnvironmentConfiguration rules) before planning a restart and return a normal failed result with errorMessage. Pre-flight the successor, or keep the predecessor alive until the successor is ready. Record helper/daemon PIDs and createdAt in .starting and treat the reservation as stale when they are dead. Release the reservation when the known rushd launcher exits without publishing an endpoint. Accept a lockfile owner that passes hello/ping. Fail fast instead of waiting out the deadline, show the last error line of <lockfile>.log, and add a recovery path (for example daemon stop --force).

This was found during an automated performance/behavior analysis of rush-client/rushd on Linux and reproduced independently by three runs with four different triggers.

Standard questions