[rush] rushd: every warm request re-fingerprints the whole workspace (sync file walk, ajv schema recompiles, sync `ps` spawns on the daemon event loop)

Start by examining the files mentioned: WorkspaceInputFingerprint.ts (lines 62-99, 221-241) for the synchronous file walk and hashing, and LockFile for the ps spawn. Look at how WorkspaceRuntimeFingerprintCache is used and where ajv schema compilation occurs (JsonSchema.ensureCompiled). Profile the daemon with --cpu-prof to see the bottlenecks in action. The fix involves memoizing file stats, caching compiled schemas, and replacing synchronous calls with async or procfs reads. Testing requires a multi-project rush workspace to measure warm request latency before and after changes.

Summary

Most of the daemon-side time for every warm request (tier-0 reuse path) goes to re-fingerprinting the whole workspace from scratch, before anything is scheduled. The cost grows with repo size, and part of the work is synchronous, so it blocks the event loop that serves all clients. After client startup (#6054) is fixed, this becomes the dominant fixed cost of a warm daemon request.

Fully warm no-op latency vs graph size (synthetic): 1.26 s at 1 project, 2.84 s at 50, 8.67 s at 300. That is about 25 ms per project, bound by re-validation. A daemon CPU profile over 12 warm requests in a 30-project workspace shows:

• lstat 466 ms and realpath 263 ms (from _hashPaths / hashFilesAsync);
• ajv schema compilation 530 ms (JsonSchema.ensureCompiled via heft-config-file: uncached project config loads recompile the schemas);
• LockFile.tryAcquire -> getProcessStartTime -> spawnSync('ps') 353 ms, about 30-54 ms per call. It runs on every request (acquireExecutionLeaseAsync), in warm-set maintenance, and in #prepareAsync.

As a result, the daemon event loop stalls for up to 190-260 ms (p99) while preparing warm requests, delaying every other client.

Repro steps

Call the same rush-lib functions that WorkspaceRequestLifecycle calls on the tier-0 path (#captureAsync at WorkspaceRequestLifecycle.ts:373 and captureProjectConfigurationFingerprintAsync at :384), with a persistent WorkspaceRuntimeFingerprintCache, 10 times against a 30-project and a 196-project workspace. Or profile the daemon (--cpu-prof) during repeated warm no-op rush-client build requests.

Expected result: Warm requests do incremental work proportional to what changed: stat-identity memoization of definition files, cached compiled schemas, and cached runtime fingerprints. Nothing synchronous or subprocess-based sits on the daemon event loop per request.

Actual result: Full recomputation on every request, with a synchronous file walk, schema recompiles, and a synchronous ps spawn.

Details

Root cause (main @ 60007c9a8c):

1. libraries/rush-lib/src/api/WorkspaceInputFingerprint.ts:62-99 (_hashPaths): a synchronous recursive readdir plus statSync/realpathSync of the ~500+ runtime files on every capture.
2. :221-241 (hashFilesAsync): re-reads and re-hashes every definition file (rush.json, common/config/**, 4 files per project) with concurrency 3, with no stat memo.
3. :199-215 (captureProjectConfigurationFingerprintAsync): uncached project configuration loads recompile ajv schemas.
4. LockFile getProcessStartTime spawns ps synchronously. On Linux, read /proc/<pid>/stat instead.

Suggested fix: memoize by file identity (dev/ino/size/mtime/ctime), making hashes a cheap stat check; cache compiled schemas process-wide; compute the runtime fingerprint once per process, since runtime files cannot change without a restart; replace the synchronous ps with procfs; and move the remaining synchronous I/O off the event loop.

This was found during an automated performance/behavior analysis of rush-client/rushd on Linux and independently confirmed.

Standard questions