Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Test-suite speed/degradation analysis across all npm test scripts (Gemini-assisted)

Open
  • Is something wrong?
#504 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
node.js, typescript
Domain
cli, performance, testing

Research direction

Start with the package.json test scripts and the CI parity check referenced in related issue #269; compare their file globs and concurrency flags against the reported runs. The analysis is explicitly unverified, so first reproduce the measurements and inspect the runner configuration. Done means a verified, scoped finding with reproducible evidence; the issue currently combines several unrelated performance, coverage, and code bug claims.

Written by the indexing model from the issue text.

Description

bug tests

Source

Full terminal logs (6 runs: npm test, test:e2e, test:unit, test:fuzz x2, test:fast; 1980 tests, 0 failures) were fed to Gemini, which compared every metric — speed, per-suite performance, degradation — and then enumerated every issue it could find. Full conversation: PALEE CLI Test Suite Analysis.

Caveat: everything below is Gemini's read of the terminal log. None of it is verified against actual code — verify before acting on any item.

Run totals

Run Tests Suites Pass Fail Skipped Wall clock
npm test 1980 297 1978 0 2 106.6s
npm run test:fast 1369 196 1367 0 2 44.7s
npm run test:e2e 160 31 160 0 0 23.0s
npm run test:unit 702 92 700 0 2 4.45s
npm run test:fuzz (run 1) 31 15 31 0 0 4.28s
npm run test:fuzz (run 2) 31 15 31 0 0 4.13s

Macro-level script comparison

Metric npm test (Full) test:fast test:e2e test:unit test:fuzz (Run 1) test:fuzz (Run 2)
Total Tests 1,980 1,369 160 702 31 31
Total Suites 297 196 31 92 15 15
Passed / Skipped 1,978 / 2 1,367 / 2 160 / 0 700 / 2 31 / 0 31 / 0
Concurrency Cap 4 (Explicit) Unbounded (CPU-1) Unbounded (CPU-1) Unbounded (CPU-1) Unbounded (4 files) Unbounded (4 files)
Pretest Build (tsc) Yes Yes Yes No Yes Yes
Wall-Clock Time 106,580.1 ms 44,726.0 ms 22,969.1 ms 4,449.5 ms 4,284.8 ms 4,133.4 ms
Cumulative Suite Time ~246,100 ms ~256,800 ms 71,134.6 ms 7,633.4 ms 6,219.5 ms 5,980.0 ms
Effective Parallelism 2.31x 5.74x 3.10x 1.72x 1.45x 1.45x
Wall Time per Test 53.83 ms 32.67 ms 143.56 ms 6.34 ms 138.22 ms 133.34 ms
CPU/IO Time per Test 124.29 ms 187.58 ms (+51%) 444.59 ms 10.87 ms 200.63 ms 192.90 ms

Headline: test:fast runs 611 fewer tests than npm test but burns ~10.7s more total CPU/IO time — Gemini attributes this to unbounded tsx worker concurrency (no --test-concurrency on test:fast/test:unit/test:e2e/test:fuzz) thrashing V8 JIT and the Windows NTFS metadata lock.

Concurrency degradation — worst CLI suites (npm test c=4 vs test:fast unbounded)

Test Suite npm test (c=4) test:fast (Unbounded) Delta Degradation
CLI adopt refuses batch filters in single-file mode (#303) 1,745.9 ms 4,024.7 ms +2,278.8 ms +130.5% (2.31x)
adoption preserves what the note already declares 4,094.9 ms 8,292.9 ms +4,198.0 ms +102.5% (2.03x)
palee adopt --undo discloses every refusal and skip (#299) 2,786.7 ms 5,628.6 ms +2,841.9 ms +102.0% (2.02x)
Roadmap import must not inherit stale depends_on_source (#223) 4,864.6 ms 9,691.7 ms +4,827.1 ms +99.2% (1.99x)
CLI Adopt declared-prerequisite edges (PAL-205 WS6) 8,881.9 ms 17,639.6 ms +8,757.7 ms +98.6% (1.99x)
roadmap import reports edges to notes that do not exist 3,405.4 ms 6,599.5 ms +3,194.1 ms +93.8% (1.94x)
roadmap import enforces the topic ID format (#308) 3,367.4 ms 6,519.3 ms +3,151.9 ms +93.6% (1.94x)
CLI config provider credentials (#82) 10,086.4 ms 19,329.6 ms +9,243.2 ms +91.6% (1.92x)
roadmap --dry-run prints the write plan (#309) 2,265.4 ms 4,299.3 ms +2,033.9 ms +89.8% (1.90x)
CLI adopt pattern accounting and repeatable filters (#312) 2,912.0 ms 5,478.2 ms +2,566.2 ms +88.1% (1.88x)
roadmap --auto-chain preserves chain head's deps (#322) 1,557.6 ms 2,884.6 ms +1,327.0 ms +85.2% (1.85x)
CLI Adopt Batch Integration Tests 6,459.7 ms 11,568.2 ms +5,108.5 ms +79.1% (1.79x)
CLI review early-warning and pillar-absence behaviour 1,861.1 ms 3,263.0 ms +1,401.9 ms +75.3% (1.75x)
CLI mastery fallback characterization 444.0 ms 775.3 ms +331.3 ms +74.6% (1.75x)
program name follows the invoked bin (#314) 3,308.0 ms 5,585.4 ms +2,277.4 ms +68.8% (1.69x)
Session CLI In-Process Coverage 2,209.0 ms 3,629.7 ms +1,420.7 ms +64.3% (1.64x)
Session hot.md read characterization 1,471.3 ms 2,398.4 ms +927.1 ms +63.0% (1.63x)
Vault target echo and PALEE_CONFIG_DIR (#311) 6,128.2 ms 9,980.6 ms +3,852.4 ms +62.9% (1.63x)
CLI config test-connection (#24) 3,029.9 ms 4,933.3 ms +1,903.4 ms +62.8% (1.63x)
Entry-point fatal output contract (#320, #338) 3,742.8 ms 5,918.5 ms +2,175.7 ms +58.1% (1.58x)
CLI Machine-Readable --json Output (INV-30) 1,234.6 ms 1,859.6 ms +625.0 ms +50.6% (1.51x)
CLI Commands 17,115.6 ms 25,176.7 ms +8,061.1 ms +47.1% (1.47x)
CLI Roadmap Wikilink + --auto-chain Integration 10,898.0 ms 14,935.4 ms +4,037.4 ms +37.0% (1.37x)
palee adopt --undo reverses adoption 23,715.3 ms 30,012.9 ms +6,297.6 ms +26.6% (1.27x)

Unit suites — peak degradation

Unit Suite npm test (c=4) test:unit (Unbounded) test:fast (Unbounded) Peak
SM-2 Algorithm 9.81 ms 47.49 ms 19.36 ms +384.1% in test:unit
loadTopics negative caching (#331) 56.69 ms 185.54 ms 125.98 ms +227.3% in test:unit
Frontmatter Parser 18.11 ms 69.13 ms 35.27 ms +281.7% in test:unit
Storage & Memory Fault Injection 158.50 ms (not in unit) 414.96 ms +161.8% in test:fast
Dependency Graph 292.52 ms 744.93 ms 583.60 ms +154.7% in test:unit
Property-Based & Fuzz Testing 27.06 ms (not in unit) 68.83 ms +154.3% in test:fast
Storage Topic Loader 120.84 ms 238.63 ms 294.02 ms +143.3% in test:fast
Storage Note Scanner 200.22 ms 310.67 ms 387.78 ms +93.7% in test:fast
TOC tier engine (PAL-205-C3) 339.57 ms 649.16 ms 566.84 ms +91.2% in test:unit
TOC discovery (storage half) 227.56 ms 217.27 ms 403.64 ms +77.4% in test:fast
Memory System 1,942.99 ms 1,722.75 ms 2,780.78 ms +43.1% in test:fast

Isolation speedups

E2E Suite npm test test:e2e Speedup
Tier 2: Boundary & Corner Cases (58 tests) 32,021.5 ms 22,305.2 ms 30.3% faster (-9.72s)
Tier 3: Pairwise Cross-Feature Interactions (16 tests) 25,029.1 ms 18,628.6 ms 25.6% faster (-6.40s)
Tier 1: Feature Coverage (53 tests) 19,414.7 ms 14,790.4 ms 23.8% faster (-4.62s)
validate --strict: e2e CLI wiring (#25) (3 tests) 1,809.5 ms 1,423.7 ms 21.3% faster (-0.39s)
Tier 4: Real-World Application Workloads (5 tests) 10,565.1 ms 8,352.8 ms 20.9% faster (-2.21s)
assessment-review independence, reverse (#40) (2 tests) 2,460.7 ms 1,852.1 ms 24.7% faster (-0.61s)
Fuzz / Stress Suite npm test test:fuzz (Run 1) test:fuzz (Run 2) Net Improvement
Challenger 1: Concurrency & Batch Stress 5,245.1 ms 3,829.5 ms 3,697.9 ms 29.5% faster (-1.55s)
Empirical Challenger 1: Deep Verification 1,423.0 ms 1,089.8 ms 1,048.1 ms 26.3% faster (-0.37s)
Property-Based & Fuzz Testing Suite 27.1 ms 20.7 ms 22.6 ms 23.5% faster
Storage & Concurrency Stress Test Suite 1,239.7 ms 1,279.5 ms 1,211.3 ms 2.3% faster
Total test:fuzz Wall-Clock N/A 4,284.8 ms 4,133.4 ms 3.5% warm-cache gain

Bottlenecks

  • 5 slowest suites = 155.8s (63.3%) of cumulative runtime: CLI Adopt --auto-chain Integration 43,669.7 ms (45 tests); CLI Migrate stored tie labels (PAL-205 #237) 36,943.5 ms (42 tests); Tier 2 Boundary & Corner Cases 32,021.5 ms (58 tests); Tier 3 Pairwise Cross-Feature 25,029.1 ms (16 tests); palee adopt --undo reverses adoption 23,715.3 ms (25 tests).
  • 35x speed gap: in-process CLI dispatch ~17.4 ms/test (104 tests in 1,809.8 ms) vs subprocess runPalee 300–1,800 ms per invocation on Windows.
  • Single-test outliers: 2,734.9 ms Windows SMB/UNC kernel timeout on \\\\server\\share (config dir guard #316); 2,028.7 ms hardcoded lock-retry sleeps in File Locking; 1,219.9 ms stale-draft 24h-clamp recovery (#328); 205.7 ms unbalanced-destination scanner stress in extractTocLinks.

Gemini's proposed fixes

  • Add --test-concurrency=4 (or 6) to test:fast and test:unit in package.json.
  • Add missing globs (validate-, provider-, engine helpers) to test:unit/test:fast — ~400 tests for <0.5s wall-clock.
  • Convert read-only/dry-run assertions in cli-adopt-autochain.test.ts (43.7s) and cli-migrate-ties.test.ts (36.9s) to in-process dispatch — ~80s to <3s.

All 8 issues Gemini found

  1. package.json script & runner misconfigs
    • 451 tests (70 suites) silently excluded from test:fast/test:unit by hardcoded filename prefixes (cli-*, engine-*, storage-*); the excluded suites are the fastest (~1.6s combined): 19 validation rules (~250 tests, ~350 ms), AI provider & network boundary (56 tests, ~505 ms), engine scheduling helpers (48 tests, ~46 ms), build/release/invariant guards (~30 tests, ~2.1s).
    • Concurrency contract violation (#269): CI matrix parity test pins --test-concurrency=4 for npm test, but test:fast/test:unit/test:e2e/test:fuzz omit it — suites run 50–280% slower.
    • test:unit has no pretest build guard; dist/-inspecting tests (palee-cli launch guard, barrel surface checks) run against a stale build.
  2. Windows performance bottlenecks
    • 2.75s SMB/UNC kernel timeout (#316): reject \\/// prefixes lexically before fs calls -> <1 ms.
    • Subprocess runPalee dominates: 3 files consume 104.3s cumulative.
    • 2.03s hardcoded backoff sleeps in File Locking: use injected/shorter test backoff.
    • 1.22s outlier in draft recovery (#328).
    • Incomplete migration: legacy allow-list of direct spawn/execFile calls bypassing the bounded runPalee wrapper (#451).
  3. Latent code bugs (pinned by characterization tests)
    • EACCES silently swallowed in session topic resolution and draft (palee session draft loses started_at instead of exit code 5) — violates the Area 2 strict-rethrow policy in memory.ts.
    • Untrimmed active_topic comparison in draft inheritance (trailing whitespace loses started_at).
    • TOCTOU inconsistency on hot.md mid session-start (vanish -> exit 5, but corrupt/vanish-tolerant paths coexist).
    • UTC-vs-local date split: session ID S-20261019T212659-... (UTC) vs Last Updated: 2026-10-20 (local) in hot.md.
  4. Test hygiene & cross-platform gaps
    • Uncaptured stdout/stderr from in-process suites polluting runner output (banners, Vault: echoes, repeated Error: EISDIR).
    • 2 Windows skips: TOC case-fold ambiguity (NTFS case-insensitive), wikilink unreadable-file (chmod 0o000 no-op on Windows ACLs).

Related

  • #451 (command invocation map — the concurrency/flag claims tie directly to its findings)
  • #269 (CI workflow matrix parity — the --test-concurrency=4 contract)
Dominant language
TypeScript
Stars
1
Forks
0
Avg merge
22h 21m
Merged PRs (30d)
167

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Kuldeep2822k/cli

All issues in Kuldeep2822k/cli

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.