Resolve native parser memory observer failures

Open
#801 15 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
typescript
Domain
backend, testing

Research direction

Start by preserving the failure evidence described in the issue, then inspect packages/parsers/src/resident_memory.ts, its tests, and the PowerShell reporting loop. Reproduce and diagnose the reader lifecycle and watchdog timing failures independently; keep the specified cadence, deadlines, limits, and fail-closed behavior. Add a failing regression before correcting a demonstrated cause, then run the affected tests and the required exact-source Windows, macOS, and Linux matrix.

Written by the indexing model from the issue text.

Description

area:parsers bug release-blocker

Parent epic: #546. Phase: P7. Critical v0.1 release blocker discovered while qualifying PR #800.

Problem and observed evidence

Exact head adb86563727119c8b4024137167d7cabe51e5a41, base dddcefc9f9a53db6147eb4f03efb3077b65b2c7d, completed all 22 expected checks with one failure: Windows verify job. The ordinary suite reported 3932 passed, 56 documented skips and one failed test: packages/cli/test/rule-semantics.test.ts, keeps a superseded source out of default results while it stays readable.

The actual error is LORE_E_PARSE_FAILED: Could not observe parser resident memory. Cause: The process memory observer stopped reporting: 5 reports; 813 ms since the last; reader 2384 (exit none, signal none); last RSS 81219584 bytes. Stack reaches packages/parsers/src/resident_memory.ts:122,169,202. The reader was still reported alive; this does not establish why reports stopped or whether the parent event loop delayed delivery. No causal fix is claimed. PR #800 remains draft and unmerged. Its passing local campaign does not override this native failure.

Raw completed log was downloaded once to /tmp/lorepack-800-adb86563-windows-verify.raw.log; readable copy is adjacent. Retain it durably with exact source and the completed check cohort. Related #566 has separate macOS reader-deadline failures; do not assume a shared cause. #677 and #783 remain closed for their verified fixes; this is a new observed failure, not a reversal of their recorded results.

Dependency DAG

Blocked by: none for source investigation and regression preparation.

Blocks #614, #618 and #623 protected merge through PR #800, and final #700/#102 qualification. The independent parser/Studio and table source work can proceed in parallel; only one local runtime verifier may execute. This is on the release critical path. Do not rerun the unchanged failed head.

Acceptance criteria

  • Retain original native log, exact source and cohort, and identify the actual reporting/reader lifecycle failure with bounded diagnostics or a deterministic reproducer.
  • Add a failing regression for the demonstrated mechanism before the production correction. Preserve genuine observer loss and oversized-input refusal.
  • Correct only the demonstrated cause. Keep the 100 ms reporting cadence, 250 ms reader/report deadline, a bounded Windows startup grace of 15000 ms, output caps, fail-closed behavior and parser memory/resource envelopes. The parser must receive no source bytes until its first valid memory sample. Do not change retries, skips, test scheduling or fixture size to pass.
  • Qualify actual affected tests, strict test types, full local verify and installed package, then a fresh exact-source Windows/macOS/Linux matrix. Retain the original resource inputs and whole-tree observations.
  • Link the correcting PR with Closes, merge only when protected checks and criteria pass, post a closing handoff and move this Project item to Done. Keep #102 open until every original release criterion is met.

Test and verification plan

Inspect monitorResidentMemory, the PowerShell reporting loop, reporting-watchdog queueing, parser lifetime and tests. First preserve the observed failure; distinguish delayed parent delivery from a stalled OS reader without guessing. Exercise successful reporting, actual silence, reader exit/malformed output, stop/cancel/cleanup, queued reports at the deadline and existing memory-bound workloads as applicable. Use real native Windows proof for Windows-specific behavior. Record exact versions, argv, source fingerprints, raw outputs and cleanup. Do not report a passing replay as an explanation of the historical failure.

Dated amendment: 2026-10-19, additional native startup failure

Fresh diagnostic source e75d74e763677b999d521d57555256505b1d0b14 also failed the terminal Windows small benchmark job. The reader never became ready: 5001ms, reader9452, no observed exit or signal, zero buffered stdout characters. This is a different phase from the original five-reports/813ms reporting gap; a shared cause is unproven. Keep both observations in this observer workstream.

The original 5000ms startup deadline and every other listed control remain unchanged. Diagnose startup and reporting independently, preserve both failures, and prove the actual mechanism before a correction or closure. The benchmark did not opt into the rule-test-only diagnostic flag, so its log has no reader timing trace. The three native verify jobs remain live; this is not a completed cohort or a green replay.

Dated amendment: 2026-10-19, Linux early reporting expiry

The exact D247 native Ubuntu verify job failed packages/parsers/test/resident_memory.test.ts:108 through the security coverage test: the reporting watchdog declared loss at measured monotonic silence 249.95140899999933 ms, while the unchanged requirement is >= 250 ms. The security suite had 92 passes, five existing skips and this one failure. Retained raw log: /tmp/lorepack-800-d247d87f-ubuntu-verify.raw.log, SHA-256 753dc0892e91eb89ce63550a4c4a843500cd92a277c655e410fad77d7e77d1a3.

This is a demonstrated premature deadline decision in the parent watchdog. It is distinct from the original Windows five-reports/813 ms reporting gap and the later zero-sample/5001 ms startup gap; no common cause or Windows correction is claimed. A timer callback is not proof that the monotonic deadline has elapsed. Keep the actual 250 ms reporting and 5000 ms startup thresholds, 100 ms sampling, fail-closed behavior and existing suite/fixture/resource controls. Add a deterministic failing regression at a measured pre-deadline time, then recheck the elapsed duration in the pending-I/O watchdog phase and schedule only the rounded-up remainder if early. Preserve report and stop cancellation. Require the original Windows failures to be diagnosed and the final exact-source native matrix to pass before closure.

Dated amendment: 2026-10-19, POSIX reader acceptance failures

The new PR #800 head 4f0cbe7462ccec44f252e6bc886ce3b26185d341 failed acceptance on two native platforms. Keep these related reader failures in this observer workstream under #546, with their distinct evidence and causes. This broadens the title to native observer failures; it does not identify a shared mechanism or waive the original Windows criteria.

  • Ubuntu acceptance: 163 passed, one failed, four existing skips. The setup build for lifecycle/diff-reads-builds-not-sources exited 2 with LORE_E_PARSE_FAILED; the nested reader errors report an aborted operation due to timeout. Source reads /proc/<pid>/smaps_rollup with the unchanged 250 ms abort signal. The log does not identify elapsed read time or why it was late.
  • macOS acceptance: 163 passed, one failed, four existing skips. The first JSON build for mixed/every-format-contributes-with-its-own-provenance exited 2 before parsing its eight inputs because the resident-memory /bin/ps reader exited with SIGKILL. Source ordering suppresses intentional monitor-stop failures. The log does not distinguish the reader's 250 ms spawn timeout from an external kill.

Both failed logs were downloaded once and retained with exact-head source under lorepack_release_evidence/windows_observer/native_4f0cbe74_acceptance_failed/retained.json. Actual synthetic merge-tree equality remains unverified at this boundary and must not be inferred from the head archive.

Next bounded step: add failure-only reader context (monotonic elapsed time, first/subsequent sample, bounded lifecycle state and original abort reason) with real failure/cancellation coverage. Keep the original cause chain, deadlines, cadence, output caps, resource envelopes and fail-closed result. A passive diagnostic is not a causal production correction. Require demonstrated failure mechanism before selecting a correction; do not blame watcher readiness or the Windows-only exact-deadline change without evidence. Source preparation can run independently; no local runtime overlaps the active full nine-bundle campaign.

Dated amendment: 2026-10-19, Windows parser-version reporting loss

Exact head 4f0cbe7462ccec44f252e6bc886ce3b26185d341 also failed the completed Windows verification job. The ordinary suite reported 3953 passes, 56 existing skips and one failure: packages/cli/test/parser-versions.test.ts, depends on a parser that contributed nothing to this project. Its build failed with LORE_E_PARSE_FAILED: four accepted reports followed by 435 ms without another report, reader 9060 with no observed exit or signal, last RSS 75939840 bytes. Stack locations include resident_memory.ts:216,250,251,136.

This is a new failing workload, distinct from the original rule-semantics five-report/813 ms loss. No shared cause is established. The parser-version test did not enable the diagnostic flag, so the retained log has no reader emission/receipt trace. Enable the existing Windows-only flag around only that test, restoring the original environment in finally. Preserve its exact files, two builds, parser-version reset, assertions, deadlines and scheduling.

The whole 4f0c cohort is now terminal and retired: 16 successes, five failures and one existing hosted-RAG skip. Its five failed job logs, all 22 conclusions and verified actual synthetic merge 39660dfa with the same tree as the head are retained and hash-checked under lorepack_release_evidence/windows_observer/native_4f0cbe74_terminal_failed/retained.json. This supersedes the earlier partial custody boundary; no unchanged-head rerun is authorized. The new trace is diagnostic evidence, not final P01 performance proof or a causal correction.

Dated amendment: 2026-10-20, macOS resident-memory query timeout

The merged release-candidate head aaff52fc7db9989bfb77f7320da8ea9878fa8476 failed macOS verify in run 39349851703, job 118391216708. The source under test was synthetic merge commit 97c48fdc27e5b9c51185d579f177ebff2e24ef4c, tree b61fb3793f3a1d34af0f45e400241c7582b7b688. packages/cli/test/resident_memory.e2e.test.ts failed bounds cmap.pdf through the shipped CLI and preserves retained builds, issue677.

The retained failure receipt shows LORE_E_PARSE_FAILED: Could not observe parser resident memory. The subsequent macOS /bin/ps read did not return within the unchanged 250 ms timeout. The process was locally killed with SIGKILL; the parent observed close after 328.349 ms. The failure context records samplePhase=subsequent, reader PID 36042, localKillRequested=true, and no OS abort reason. Before fail-closed cleanup, sampled parser RSS reached 1,065,762,816 bytes, about 1,016 MiB, below the 1,024 MiB threshold. The 1,081-byte cmap.pdf fixture and exact receipt are retained at lorepack_release_evidence/windows_observer/native_aaff52fc_macos_reader_timeout/; receipt SHA-256 is 667d2653fafeff9b58d171916aaf703d2f38d106e433e711d1d7c18d2bed4ca6.

This proves the observer's own per-sample query timed out and was killed during this workload. It does not establish whether process launch, ps query execution or host scheduling consumed the deadline, and it does not explain the distinct Windows startup/reporting failures. Preserve the original 100 ms sampling cadence, 250 ms reader/report deadline, 5000 ms startup deadline, 1,024 MiB threshold, fixture bytes and test scheduling. Do not change the test to accept this typed observer failure in place of the required resource refusal. The other jobs in run 39349851703 remain live; no terminal cohort or correction is claimed.

Dated amendment: 2026-10-20, Windows reporting deadline on the corrected release candidate

PR #807 head 93724be38115bf9e94ba956db25d8e68ea4b9abc, source tree b61fb3793f3a1d34af0f45e400241c7582b7b688, completed run 39354760020. macOS and Ubuntu verification passed. Windows failed packages/cli/test/delivered_context_budget.test.ts, bounds complete markdown output including omissions and framing, issue424, after 4,666 passed tests, one failed and 63 existing skips.

The typed observer error followed nine reports and 254.483 ms since the last report against the unchanged 250 ms deadline, 4.483 ms over. Reader 6732 remained alive without an observed exit or signal; last RSS was 108,756,992 bytes. Failure-only owner context measured 254.631 ms total, timer fire at 254.270 ms, 0.361 ms from timer fire to check, 197.083 ms owner-active time and 57.549 ms idle time, with event-loop utilization 0.774. This distinguishes the report-deadline failure from the macOS ps reader timeout above, but does not establish why the owner interval ran late or prove a shared cause with the historical Windows failures.

The full Windows job log and test artifacts are retained under lorepack_release_evidence/windows_observer/native_93724be3_reporting_gap/. Manifest SHA-256: b04aec4a85f8946ad3c1a14265c1535e83eac3db32c7807f2c4f8fa36e3f9349; raw job log SHA-256: 3ed62d668b29e90c8c288571d5bd6480955297a0be79dd1872b9520bdfe8091f. Keep the exact 250 ms reporting guard, 100 ms cadence, fixture and assertions. Diagnose the measured late interval before a production correction; do not turn this observer error into an accepted outcome for the original resource-bound case.

The earlier run 39349851703 is now terminal: macOS failed the cmap.pdf observer case recorded above, while Ubuntu and Windows both failed only because the candidate combined a changeset with a [no release] commit marker. The marker has since been removed from the test-only commit message without changing the source tree, and a new verification is running on the corrected head. Neither failure has been treated as a pass.

Next bounded step: run the exact Markdown-budget case with the existing failure-only Windows trace, then compare the reader's sample-phase events with parent receipt and watchdog timing. The test-only diagnostic does not correct or waive the 250 ms reporting failure.

Dated amendment: 2026-10-20, idle-owner Windows reporting gap

PR #807 head 0e071b13296506406f7bb8bc56d5090bf2933f36, source tree 846a9a809bafdcf34463312f438169cad5ae8541, completed run 39358353199. macOS and Ubuntu verification passed. The Windows run passed the diagnostic-enabled Markdown-budget case, then failed packages/cli/test/rule-semantics.test.ts, ranks a higher declared authority above a lower one, after 4,666 passed tests, one failed and 63 existing skips.

After one report, the observer reported 576.890 ms of silence against the unchanged 250 ms deadline. Reader 1136 remained alive without an observed exit or signal; last RSS was 64,196,608 bytes. Owner context measured 577.073 ms, timer fire at 576.700 ms, 0.373 ms from fire to check, 0.743 ms owner-active and 576.328 ms idle, utilization 0.001. This indicates that parent event-loop congestion did not account for this occurrence. It still does not distinguish a PowerShell read stall, process scheduling, or pipe delivery. The failing test had not enabled the detailed phase trace.

Retained evidence is under lorepack_release_evidence/windows_observer/native_0e071b13_rule_semantics_gap/. Raw Windows job log SHA-256: bff91a79ec509827ee1c1069a10bfa21b1df0cf180605ddb7a9555ac87ef07dd; owned inventory SHA-256: 3500f53b84e475e40cf6484e103dcbb977a242832d428714a8519a9918112b2d.

Next bounded step: enable the existing bounded Windows diagnostic only for this exact rule-semantics case and use the reader milestones and parent receipt events to locate the gap. Preserve the reporting threshold and original assertions. The earlier delivered-context case remains a separate failure that passed on this run; no common cause is established.

Dated amendment: 2026-10-20, cmap PDF reporting gap recurred on Windows

PR #807 head 8d7cb766a9b52c55c53c32fe19c18cefa99e38fc, source tree c461b0a2b4ce30ab35229cff2a395cba05dc7389, completed run 39363026869. macOS and Ubuntu verification passed. Windows passed the diagnostic-enabled rule-semantics test, then failed packages/cli/test/resident_memory.e2e.test.ts, bounds cmap.pdf through the shipped CLI and preserves retained builds, issue677.

The retained receipt records the 1,081-byte fixture, unchanged 100 ms sampling, 1,024 MiB resident threshold and 1,536 MiB observed envelope. Maximum sampled parser RSS was 1,079,222,272 bytes. Test stderr says cmap.pdf exceeded the 1,024 MiB limit and was left out. The measured command exited 2 and its captured stdout contains LORE_E_PARSE_FAILED caused by one report followed by 491.276 ms silence, with the 250 ms deadline exceeded by 241.276 ms. Reader 6368 had no observed exit or signal and last RSS was 62,021,632 bytes. Owner context measured 0.582 ms active, 490.884 ms idle, utilization 0.001. The exact receipt does not establish which of the two parser PIDs produced the limit warning or observer error, so do not collapse them into one process-level cause.

The detailed bounded trace was enabled for the rule-semantics test on this run; that case passed. The cmap.pdf test remains uninstrumented and failed. Raw log, receipt and owned inventory are retained at lorepack_release_evidence/windows_observer/native_8d7cb766_cmap_observer_gap/. Manifest SHA-256: e64720bd4487934d5d47698de318679f581ae6c203e4ffb2f05ee2dbdbb7096b; raw log SHA-256: 1a221835b69babc1e8091dc60d583b2b3b1bbf977b6720bd66dd523a56281e12; fixture receipt SHA-256: 1b7c458da1c31ea7af1a2e128c6ad3b0f73edb4e31aee4c30beb86df46987cbe.

Next bounded step: enable the existing Windows-only failure trace on the exact cmap.pdf case. Preserve the fixture, 100 ms cadence, 250 ms observer deadline, 1024 MiB threshold, 1536 MiB envelope and retained-build assertions. A failure-only trace is diagnostic, not a waiver or correction.

Dated amendment: 2026-10-20, Flate PDF reporting gap on Windows

PR #807 head cd796de5ca5dc5c049c21ab202f654d7b45b5f39, source tree bfc2b96ed7b94d5747656a8f5ace13f3563f0b2a, completed run 39367684047. macOS and Ubuntu verification passed. Windows passed the trace-enabled cmap.pdf case, then failed the separate flate.pdf case in packages/cli/test/resident_memory.e2e.test.ts.

The flate.pdf receipt records 3,815,885 fixture bytes, maximum sampled parser RSS 732,880,896 bytes, no remaining parser processes and a typed observer failure after 48 reports. The reader was still alive without observed exit or signal. The last report gap was 526.421 ms against the unchanged 250 ms deadline. Owner context measured 804.094 ms total, timer fire at 289.385 ms, then 514.709 ms from fire to watchdog check; active time was 557.998 ms and idle time was 246.098 ms. The observerError receipt field is null because the command returned the typed error in its captured JSON stdout; process exit was 2. No memory-threshold warning occurred for this fixture. The failure is distinct from the cmap.pdf threshold warning and observer gap in the prior run.

The existing detailed trace was limited to cmap.pdf; this flate.pdf failure has only owner context. Retained log, receipt and inventory: lorepack_release_evidence/windows_observer/native_cd796de5_flate_reporting_gap/. Manifest SHA-256: 960e33f68aba62ed0b96b7621dceeccbe120bb82d107cd2bd21b4501ae34077f; raw log SHA-256: e98f760b0df7f2c662ecec749d7f5c19087e3e3bdf020994a4c323abd79babf9; fixture receipt SHA-256: 42a9c8a6e4774c1d698ffddbc2561d08b20ed74a81a07fd6d23e99b530bb752f.

Next bounded step: enable the existing Windows-only bounded trace for both original PDF cases. The traced cmap.pdf pass does not prove a correction. Keep the existing fixtures, resource envelopes, report deadline, cadence and retained-build assertions.

Dated amendment: 2026-10-20, auxiliary WMI target-exit classification race

PR #807 head 4e7001e14c924e326f7486a95bdfb8840deb7ebf, synthetic merge c79ca79dde253bee42f8e5d2271052976ba3881d, source tree 64e96e71ef10273dc4c5fbc704f183dd8519c709, completed run 39372753376. macOS and Ubuntu passed. The Windows PDF memory suite passed both original PDF fixtures on this run. Windows full verification failed only the auxiliary WMI comparison test at test/resident_memory.test.ts:773.

The WMI control's native growth check succeeded: blocked growth lasted 859.126 ms, and an independent snapshot grew from 62,767,104 to 199,471,104 bytes. The wrong-identity reader exited with code 1 and emitted its identity diagnostic. After the target was killed, the candidate WMI reader also exited with code 1 and emitted LPF:get:80041002, but the 250 ms reporting watchdog classified the promise first: last report at 2,537.510 ms, failure at 2,789.084 ms, 251.574 ms of silence. The reader had no signal. This is an ordering race between a valid report-deadline failure and the auxiliary reader's close, not proof that the production PDF cases failed on this run. The candidate test must still prove the WMI reader observes target exit and closes cleanly.

Raw log, test artifact and manifest are retained under lorepack_release_evidence/windows_observer/native_4e7001e1_wmi_exit_race/. Manifest SHA-256: 214e8fcf7b2235ea6f9892df36f1f394eef6f8c23ba5aca36d8c77cbe1e91ccd; raw job log SHA-256: df563bc2041e0094f4c905bb9c9ed0e81cbf40358bf97e570284e9217682f320.

Next step: fix the auxiliary assertion so it preserves both required proofs: the target-exit WMI error and natural reader exit, while allowing the 250 ms watchdog to win if the close arrives just after its deadline. A report-deadline followed by a killed reader without LPF:get and exit code 1 must continue to fail.

Dated amendment: 2026-10-20, macOS subsequent reader timeout and trace redaction

PR #807 head 734c30491dcfaba54427e1d255d74b987796c1a6, run 39409870046, failed macOS packages/parsers/test/resident_memory.test.ts, stops native resident growth outside a blocked parser, issue677. The subsequent /bin/ps read took 277.089 ms against its 250 ms timeout and exited with SIGKILL. The cohort reported 4719 tests passed, one failed and ten skipped. Raw job log and receipt are retained at lorepack_release_evidence/windows_observer/native_734c3049_macos_issue677_reader_timeout/; raw log SHA-256 a74f927c97a30477f8d174a6a46e3bd090a96a0ddf3e5a9a61bdd9072a98d648. This is a real bounded-reader failure and does not justify a deadline increase or a test skip.

A separate Windows acceptance failure on run 39411016910 had two scenarios fail after 128 reports and 435 ms of silence while the reader remained alive. Its raw log is retained at lorepack_release_evidence/windows_observer/acceptance_126b9079_windows_failed/job.log with SHA-256 c5f889acdf362c23a491f6bae831abb0383c30fbf0c839d84495e48d87eef0d8.

Inspection found that CLI secret redaction treated PublicKeyToken= inside the bounded observer JSON as a secret pair and consumed the rest of that quoted JSON value. A focused test reproduced the malformed rendered trace. A local fix on branch 801-observer-diagnostic-redaction preserves the quote boundary while still redacting the token, and adds regressions for JSON preservation, apostrophes and quoted values. This explains the missing diagnostic suffix, not the underlying reader gaps. The full Windows failure trace after the redaction fix is still needed before claiming a native cause.

Keep the parser's sampling cadence, deadlines, fail-closed behavior, fixture size and resource limits unchanged. Continue to diagnose the Windows PowerShell reporting gaps and POSIX reader timeouts as distinct events unless new evidence establishes a shared mechanism.

Dated amendment: 2026-10-20, two distinct Windows verify failures

PR #807 head 734c30491dcfaba54427e1d255d74b987796c1a6, run 39409870046, completed with two Windows test failures after 4665 passed and 63 skipped. packages/cli/test/delivered_context_budget.test.ts, JSON output case, missed a report at 303.513 ms against the 250 ms threshold. Reader 7040 remained alive. Owner time was 218.680 ms active, 85.027 ms idle and 0.72 utilization; 34.832 ms elapsed between timer fire and check. Parent event-loop activity may have contributed to this case.

packages/cli/test/retrieval.test.ts, rollback ranking fixture, independently hit its 5000 ms startup deadline at 5196.748 ms with reader 7240 alive and no output. Owner time was 8.298 ms active, 5188.466 ms idle and 0.002 utilization. Parent event-loop congestion does not explain this startup case. Do not conflate the two failures.

Raw Windows and macOS logs plus receipts are retained at lorepack_release_evidence/windows_observer/native_734c3049_verify_failures/. Windows log SHA-256 98a80f4e80687b48f7f0e76b039ff4976853178438d41a8aa6841552e3e72004; macOS log SHA-256 a74f927c97a30477f8d174a6a46e3bd090a96a0ddf3e5a9a61bdd9072a98d648.

The next diagnostic PR sets the existing bounded trace only for the affected Windows delivered-context JSON and rollback retrieval tests. It leaves their assertions, sampling cadence, deadlines and resource inputs unchanged. The core redaction fix preserves trace JSON after redacting assembly public-key tokens. Wait for the exact-head Windows traces before selecting a production correction.

Dated amendment: 2026-10-20, Windows observer startup silence on benchmark

PR #817 benchmark run 39435155137, source e72b5b315602ec800a1b4ec21a45e2ebc84d5405, failed the Windows small benchmark at the unchanged 5000 ms startup deadline. The reader process 8228 remained alive, stdout had zero buffered characters, and the bounded trace contained only startup-arm at parent millisecond 1, spawn at 4, and startup-fire plus startup-check at 5003. There was no reader close, process identity, PowerShell script-entry or sample event.

This is a distinct startup silence, not the 265 ms report gap in the retained-build case, the macOS subsequent ps timeout, or the long-history test timeout in #816. The trace does not establish whether PowerShell initialization, pipe delivery or Windows scheduling caused the silence. Do not increase the startup deadline or classify the observer as fixed from a later pass.

Raw job log and receipt are retained at lorepack_release_evidence/windows_observer/benchmark_startup_e72b5b31/. Raw log SHA-256: b6527c303591dc6f20c7c40e430acbcbbd1d9226b2499674713e247277910f77.

Next step: use the exact-source Windows traces to isolate PowerShell process startup from pipe delivery while keeping the 5000 ms startup deadline and fail-closed behavior unchanged.

Dated amendment: 2026-10-20, installed Windows package startup silence

The expanded PR #819 run 39449531110 failed its Windows packed-install check, job 118720323737, on source 73b5a417995e62b2583676fb26238e3313e2345a. After npm installed the exact 145-package production tree, the installed CLI began building a 73-file project and failed closed with LORE_E_PARSE_FAILED: its PowerShell reader was still alive at 5002 ms, with zero buffered stdout, no exit code and no signal. The check did not set the bounded observer diagnostic flag, so this log does not locate the reader's startup stall. The failure is consistent with the earlier Windows benchmark startup silence, but does not prove a shared cause.

The raw job log is retained at /tmp/lorepack-packed-install-windows-39449531110.zip, SHA-256 80d18053c86894ac4fb958115e73a2aa6f3afdb1238c69bb269962e3c702b022.

Next step: capture the bounded observer trace from this exact installed-package Windows reproduction, then use its process and script-entry evidence to distinguish process startup from command execution and pipe delivery. Keep the 5000 ms startup deadline, 100 ms cadence, fail-closed behavior, package check and build fixture unchanged. Do not retry the unchanged check as a substitute for diagnosing it.

Dated amendment: 2026-10-20, manual packed smoke passed under diagnostics

Manual Windows observer run 39450999465 completed successfully on source 27d300901f56d28bd33347f483792421ca106a81, Node v24.21.0, pnpm 11.18.0. The exact packed install smoke took 163 seconds, installed the reviewed 4992-member tarball and 145-package production tree, built and searched the PDF without loading native modules, and passed the full packed CLI assertions. The existing large-result pipe acceptance case also passed.

This single instrumented pass does not correct or explain the prior uninstrumented 5002 ms startup failure. The successful packed smoke emits no observer failure trace, and runInstalledCli returns stdout without forwarding the captured child stderr on success. The retained raw packed log SHA-256 is fba0038f2a0ec7d771d5bfaf020809231e89a271daafcd6e9e9259801e1651d5. Run 39450999465 retains the exact source identity and artifact hashes.

Keep #801 open. The next exact failure in the manual diagnostic path will include the bounded observer trace in the captured error output. No shared mechanism, production correction or deadline change is claimed.

Dated amendment: 2026-10-20, D1 test reporting gap on Windows

The expanded PR #819 Windows verify run 39449531110 also failed packages/deploy-cloudflare/test/query-budget.test.ts, test discovers a cited table within the original context query maximum, issue429. The observer had emitted one report, then remained silent for 882 ms; reader 2988 was still alive and last RSS was 66,531,328 bytes, with no exit code or signal. This is a reporting gap after startup, distinct from the zero-output startup failure in the same run's packed-install check and earlier Windows benchmark. Do not infer a shared mechanism.

The raw Windows verification log is retained at /tmp/lorepack-pr819-windows-verify-39449531110.log, SHA-256 ec69c957ef07947d409badf9476f5187cab846f1b7a689437cdf754a8790d41c.

PR #820's manual Windows observer qualification is being extended to run this exact D1 query-budget test with diagnostics enabled, alongside the packed install workload. The exact case passes locally on macOS, but no Windows trace has yet been captured for it. Preserve the original test assertions and observer limits. Use the bounded trace to distinguish a stalled PowerShell sample from delayed report delivery before proposing a production correction.

Dated amendment: 2026-10-20, macOS observer query timeout during cancellation test

PR #820's macOS verification on source 27d300901f56d28bd33347f483792421ca106a81 failed packages/cli/test/dev.e2e.test.ts, leaves the active build intact and releases the build lock. The follow-up build exited 2 instead of 0 after the cancellation scenario. Its cause reports a Darwin resident-memory query in the subsequent phase taking 516.476 ms; the /bin/ps reader PID 19824 had no exit code and was killed locally after the 250 ms query deadline. This is a POSIX reader timeout during a high-workload cancellation integration, distinct from the Windows PowerShell startup and report gaps.

The raw macOS verify log is retained at /tmp/lorepack-pr820-macos-verify.log, SHA-256 7a6720f3a900a130b0de2ad6995c1208dc9c5fe1d231f5b94398deb646d1a229. The suite had 4687 passes, 9 skips and this one failure. The failure's structured diagnostics include the cancellation stages and subsequent reader context; they do not identify why ps exceeded its deadline.

Keep the 100 ms sample cadence, 250 ms reader deadline and cancellation test workload unchanged while investigating. Do not infer that this macOS query timeout shares a cause with the Windows PowerShell failures.

2026-10-20 local reproduction: cancellation case passes three times

On the same macOS host used for development, the exact packages/cli/test/dev.e2e.test.ts cancellation case passed three consecutive focused runs. Each took 54.58 to 55.69 seconds. A separate 100-call /bin/ps -o rss= -p <pid> probe measured 1.61 ms median and 2.92 ms maximum. This confirms the remote 516 ms observation is not reproduced under ordinary local load; it does not validate behavior under CI contention or explain the timeout. Keep the failure open and preserve the original deadlines while tracing the loaded-runner condition.

2026-10-20 manual diagnostic run: D1 case passes, packed check stops at audit

Manual Windows observer run 39460644881 ran the exact D1 query-budget test on source 4026773d3b4d7cdc519185eda5b7ef791c210901, Windows, Node v24.21.0, pnpm 11.18.0. With LOREPACK_OBSERVER_DIAGNOSTIC=1, the D1 case passed in 4366 ms; two sibling tests were intentionally filtered by the targeted command. Its retained JSON says 1 passed, 0 failed, 2 skipped. The raw D1 output SHA-256 is e00bd56672f30b74310095bebfabbe7861a15197325d7132d5714c7998c1b2b2b.

The same run's packed install smoke built and installed the 145-package production tree, then completed its parser, query, import/export and CLI assertions. The final packed-package npm audit --omit=dev rejected the old Hono dependency with two moderate advisories. It did not fail in the resident-memory observer and produced no observer failure trace. Issue #821 tracks the Hono update. This exact run does not explain the prior Windows startup silence.

2026-10-20 acceptance failure isolation

Downloaded the completed Windows acceptance job log directly from job 118758843469 in workflow run 39460625520 (job log response was available before the overall workflow finished). The failing scenario returned during setup, lorepack build --allow-large-project, before its inspect sources --json step. The setup builds 2,501 generated documents, the same large count used by the separate scale scenario. Therefore this failure does not exercise or falsify the pipe-output assertion itself.

The retained trace shows owner event-loop activity of 918.542 ms and 99.9% utilization during the 919 ms watchdog window. It records before-rss at reader time 13,853 ms but no later phase marker. Since the owner event loop processes both stdout reports and stderr diagnostic markers, the missing marker does not prove that WorkingSet64 itself stalled. Parent event-loop starvation is now a strong competing explanation; independent reader-side timing is still needed before assigning causality.

The pipe scenario remains valuable because it verifies the actual CLI result exceeds the historical 65,536-byte truncation boundary and is complete JSON through a pipe. Its setup should be reduced to the smallest fixture that remains comfortably above that boundary, with an explicit byte-count assertion. Keep the separate 2,501-document scale scenarios unchanged, since they assert above-envelope build and read behavior. Validate the test's actual command and byte count on all three operating systems before considering this correction complete.

Dated amendment: 2026-10-20 macOS full-suite reader deadline

Two independent macOS full-suite jobs failed in packages/cli/test/parse-limits.test.ts while building a nested-markdown fixture. The current main CI run 39481151862 and PR #826 CI run 39481239541 each report Could not observe parser resident memory. Their POSIX reader diagnostics show a subsequent /bin/ps query locally killed at the unchanged 250 ms timeout: 273.576 ms on main and 294.299 ms on PR #826, both with signalCode: SIGKILL and localKillRequested: true. Neither trace establishes why the query exceeded its deadline. The exact focused file passed 26/26 on local macOS in 17.67 seconds.

Treat these as two new macOS observations in the resident-memory reader lifecycle, distinct from the Windows startup/reporting traces. Preserve the 100 ms reporting cadence, 250 ms reader/report deadline, 5000 ms startup deadline, output caps and fail-closed behavior. Diagnose the reader delay under native CI conditions before proposing a correction. Do not attribute it to the concurrent runs without evidence.

A further same-day Windows observation is available from PR #813 run 39411654349, Windows verify job 118590626815. The real CLI resident-memory test for flate.pdf failed closed after 30 reports and 265 ms without a report. The observer reader was still reported alive, with exit and signal unset, and its last RSS was 568197120 bytes. This does not establish whether report delivery stalled, the reader stalled, or the owner process was delayed. Preserve this as a separate Windows report-gap trace alongside the two macOS /bin/ps deadline traces.

The earlier PR #817 benchmark run 39435155137, Windows job 118670127555, also preserves a startup-phase trace: the observer spawned at parent time 4 ms, reached the unchanged 5000 ms startup deadline, and had emitted no identity or output. No reader close, exit or signal was observed by the deadline. This is another Windows startup observation, separate from the 265 ms report gap and the prior exact-run startup failure. It does not establish a shared cause.

Current main run 39481151862, Windows acceptance job 118829753642, also failed the real CLI startup for init/credential-shaped-files-never-indexed: the observer did not become ready after 5001 ms, with a live reader and no buffered output. The later build-not-found outputs are consequences of the failed initial build. This is the acceptance suite exercising its credential filename warning and exclusion guarantee, not evidence that the scenario is redundant.

Diagnostic instrumentation prepared on the #820 branch

Commit f77cced0 now includes the bounded stdout and stderr buffers in macOS reader exit errors; each stream already has a 256-byte cap. A new real-child test first failed because a complete 312\n RSS line was available but the child remained alive until the original 250 ms timeout, and the error discarded that line. With the diagnostic change, the same scenario retains the line in the error. This demonstrates a case the reader can distinguish, but does not prove that either hosted ps query produced output before its timeout. The reader behavior and all deadlines remain unchanged. The full parser suite passed 321 tests with five documented skips on local macOS; the focused macOS/POSIX reader tests passed 27/27, and lint, format, typecheck and test-type audit passed. Push and rerun only after the active verification cohort clears so the next hosted trace can determine whether output was present.

Dated amendment: 2026-10-21, macOS security-suite recurrence

PR #819 run 39567942326 completed with Ubuntu verification passing, Windows verification failing one unrelated 5-second DOCX test deadline, and macOS verification failing in the consolidated security suite. On macOS, test/security-coverage.test.ts reported that packages/cli/test/parse-limits.test.ts, a file that exceeds a parse limit leaves out brackets.md with parse-timeout, and activates without it, failed rather than passed. The parser error was LORE_E_PARSE_FAILED: Could not observe parser resident memory. The stack reaches resident_memory.ts:247 (fail) and resident_memory.ts:526 (poll). The security suite had 93 passes, 8 existing skips and this one failure; its Vitest-reported test time was 402.72 seconds.

This reproduces the macOS observer failure class already tracked here, but the retained output does not identify whether /bin/ps timed out, exited, or returned invalid output. Do not infer a common cause with the Windows reporting/startup failures. Keep the 100 ms sampling interval, 250 ms reader deadline, 5000 ms startup deadline, fail-closed behavior, and test workload unchanged until the reader failure is identified.

Scope amendment: bounded macOS reader diagnosis, 2026-10-21

The #819 run identifies a repeated macOS Could not observe parser resident memory failure but its assertion output does not retain the nested /bin/ps reader cause. Extend the existing manual diagnostic so one macOS run of the exact parse-limits regression retains bounded reader exit, signal, timeout and error-code evidence. Keep normal runs unchanged and keep all reader deadlines and parser assertions unchanged. Use the focused test on macOS rather than rerunning the full three-platform matrix. A passing replay still does not explain the failure.

2026-10-21 scope amendment

The macOS security coverage case from run 39567942326 failed in the nested parse-limits.test.ts execution, but its assertion output did not retain the observer's nested /bin/ps cause. A direct focused replay and a replay through the same security coverage path both passed, so the cause remains unproven. The diagnostic branch now runs the actual security coverage path and forwards up to four bounded observer failure lines from its child process. Preserve the existing test scheduling, reader deadlines, workload and assertions until a reader failure mechanism is captured.

2026-10-21 Windows startup deadline amendment

Changed-source PR #835 produced three Windows PowerShell startup failures across verify, clean-install and benchmark contexts. Two retained traces show the reader process was spawned, remained alive, and emitted no PowerShell entry marker, process identity or RSS within the original 5000 ms deadline. The third failed with the same empty-reader state. This locates the immediate failure at process/command startup before the memory query begins. It does not establish whether runner scheduling or PowerShell initialization caused the delay.

Amend the startup bound to 15000 ms so hosted Windows startup latency does not become a parser failure. This does not grant the parser additional unobserved work: isolation sends no source bytes until both parser readiness and the first valid memory sample. Keep the 100 ms sampling cadence, 250 ms reporting deadline, output bounds, memory envelope and fail-closed behavior unchanged. Add a regression where the first valid sample arrives after 5000 ms but before 15000 ms. Preserve a test proving startup still fails at the new deadline.

Dated amendment: 2026-10-22, unused-parser identity test

The next exact-head run 39761048482 passed Ubuntu and macOS verification but Windows failed in packages/cli/test/parser-versions.test.ts, depends on a parser that contributed nothing to this project. The build's PowerShell memory reader emitted one valid sample and then had no report for 623 ms against the unchanged 250 ms watchdog. The reader was alive when checked; the owner event loop utilization was 0.001. The trace did not show the reader reaching its next after-sleep marker. This is a new Windows reporting gap, not evidence of the original large-build stall mechanism.

That test asserts that changing an unused parser version changes the real build ID. Its one-Markdown-file fixture also launched an unrelated parser process and its memory observer. The test now uses a source-free project while retaining two real runBuild calls at separate temporary paths, the parser-version mutation and restoration, and the build-ID inequality assertion. This is a scoped exception to the earlier exact-file requirement for this identity test: an unused parser contributes no parsed files, and parsing Markdown does not strengthen the identity assertion. The other observer workloads, limits and assertions remain unchanged.

Focused parser-versions.test.ts passed 6/6 locally; Biome and changeset policy passed. Exact candidate 0a801d9312b3be0680d8be990732c3370e5c79fb is under its required Windows, macOS and Ubuntu matrix 39768645876. Closure still depends on the native observer criteria and every original issue acceptance item.

Dominant language
TypeScript
Stars
2
Forks
1
Avg merge
5h 36m
Merged PRs (30d)
221

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from burakdede/lorepack

All issues in burakdede/lorepack

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.