ILSpy.Tests wall clock: measurements, sharding speed-up, and in-suite waste (2026-10-18)

Start with the named test files, especially AnalyzerResultHighlightTests.cs and ReferenceClickTests.cs, and compare their setup with the faster fixture-based examples identified in the issue. The issue also points to NavigationHistory.cs and the Linux Desktop workflow for later cleanup and sharding work. Done means applying the proposed changes in order and measuring the suite again; the issue gives baseline timings and target savings, but covers many files and several independent changes.

Machine: a 24-logical-CPU Windows workstation, Debug build of HEAD 6d5b5fe09, suite run alone from
ILSpy.Tests/bin/Debug/net11.0/ILSpy.Tests.exe --report-trx. All numbers below come from the
TRX files and process-tree sampling of the test host.

1. Measured

Single process (the status quo)

run	wall	tests	CPU time	peak working set
baseline 1	370 s	1348 passed + 4 skipped	n/a	n/a
baseline 2	358 s	same	n/a	n/a
baseline 3	378 s	same	458 s (1.2 cores)	1.4 GB
earlier run under dotnet test --solution (all hosts concurrently)	529 s	same

Wall clock equals the sum of test durations (inter-test gap 4.5 s total): the suite is strictly
serial. Noise band about 20 s (5 %). The solution-level run is 40 % slower purely from contention.

Why it is serial, and stays serial in-process

• [assembly: AvaloniaTestIsolation(PerAssembly)] creates one HeadlessUnitTestSession
  (s_session: Dictionary<Assembly, Session>), and AvaloniaTestMethodCommand.Execute blocks on
  session.DispatchCore(...) onto that session's single dispatcher thread (decompiled
  Avalonia.Headless.NUnit 12.1.4 with the repo's ilspycmd). NUnit workers would just queue.
• App state is process-global on purpose: AppComposition.Current,
  ILSpySettings.SettingsFilePathProvider, the window list in ResetAppStateAttribute.

The only parallel option is running N test-host processes with disjoint filters.

Process-level sharding

Filters: FullyQualifiedName~ICSharpCode.ILSpy.Tests.<Namespace>. (and <Fixture>. for the
root namespace, split greedily by fixture totals). All shards green; 1349 tests reported, the 3
missing rows are [Explicit] tests that a filtered run does not list at all.

config	wall	speed-up	per-shard wall	per-shard CPU	per-shard peak WS	sum of test durations
1 process	369 s (mean)	1.0x		458 s	1.4 GB	367 s
2 shards, run 1	220 s	1.68x	218 / 217 s			432 s
2 shards, run 2	234 s	1.58x	230 / 227 s	284 / 276 s	3.1 / 1.1 GB	454 s
4 shards	141 s	2.62x	109 / 129 / 95 / 138 s			468 s

Observations:

• Each test runs 20-30 % slower under concurrency even with 20 idle cores (sum of durations 367 s
  -> 454 s -> 468 s). The shared resource is not CPU; most likely disk/page cache for the CoreLib
  metadata every test reloads, plus GC. Speed-up is therefore sub-linear.
• 4 shards are unbalanced (95 to 138 s) because the balance was computed from single-process
  times; a rebalance would bring 4 shards to roughly 120 s.
• The Analyzers + TextView + Bookmarks shard peaks at 3.1 GB because those fixtures build
  CoreLib type systems; a 4-core / 16 GB CI runner can hold 2 shards comfortably, 3 at most.
• dotnet test --project ILSpy.Tests/ILSpy.Tests.csproj --no-build --filter "FullyQualifiedName~..."
  passes the filter through to the NUnit bridge (33 Themes tests ran in 6 s), so CI needs no
  special launcher. Only the --treenode-filter / --filter-class forms are rejected.

CI projection (Linux Desktop job, Release, "Execute UI tests" 269 s today, 4 cores):
2 shards as concurrent background jobs in the same step should land near 165 s, saving
about 1.7 min of a 14 min job; the decompiler tests that follow (6.3 min) remain the long pole.
macOS (3 cores) similar. The Windows job already runs ILSpy.Tests concurrently with the other
hosts and is bounded by the decompiler suite, so sharding gains nothing there.

Harness knobs (measured as no-ops)

Each was a one-line scratch edit, one full run, then reverted (working tree is clean).

experiment	wall	verdict
Waiters.PollInterval 25 ms -> 5 ms	357 s	inside the 358-378 s noise band
skip AppComposition.CreateContainer() for non-[AvaloniaTest] tests (793 plain tests)	354 s, all green	inside the noise band

So polling granularity and the per-test container rebuild are each worth at most ~10-15 s and
cannot be distinguished from noise. The time is inside the tests.

Where the time is (mean of the 3 baselines, 367 s)

bucket	tests	seconds
< 0.1 s (plain tests, no UI boot)	787	27
0.1 - 1 s (boot + a small action)	464	145
1 - 2 s	71	95
>= 2 s	30	100

Calibration anchors: boot-only test 0.30-0.36 s; Ctrl+R analyze of one member 0.63-0.70 s;
decompiling System.String or System.Object 1.6-2.5 s; a whole-Enumerable decompile 4-6 s.

2. Findings (estimated, from TRX deltas and code reading; file:line as of HEAD)

A. Slow outliers that decompile framework types only to get "a document" (~75 s of 367)

The pattern: select System.String / System.Linq.Enumerable / System.Object, wait for the
full decompile (or let teardown's DrainPendingWork, capped at 10 s, absorb the cancelled one),
when FixtureAssembly.OpenFixtureAsync (Greeter/Widget, 0.2-0.5 s) would satisfy the assertion.

test(s)	measured	note
Analyzers/AnalyzerResultHighlightTests.cs (2 tests)	9.1 + 4.5 s	String decompile, then ActivateItem reselects a String method and HighlightLocalReferences walks every reference; neither test needs String
Bookmarks/BookmarkAnchoringTests.cs:50-68	5.9 s	synchronous DecompileType(Enumerable) and CreateForLine on every line, no early exit
Docking/SetActiveDockableTests.cs:46	5.5 s	Enumerable just to have an active document
Commands/CommandLineArgumentsTests.cs:84	5.0 s	selects the Enumerable type node and awaits its decompile so ActiveDecompilerTab is non-null; the sibling test at :102 does the same navigation in 0.4 s
Bookmarks/BookmarkNavigationViewTests.cs (3 tests)	10.2 s	String x2 + Object; String; Object + the CoreLib TypeDef table
TextView/ReferenceClickTests.cs:54 (5 tests)	14.6 s	5x String, while MemberHighlightSample in MemberReferenceHighlightTests.cs:76-89 runs at 0.35 s/test; :152, :192 of that file also use String (5.9 s)
Editor/TabFocusNavigationTests.cs:52 (2)	6.1 s	String only to focus the editor
TextView/ILLanguageTooltipTests.cs:126,156	6.0 s	String decompiled in C# just to get a tab
TextView/DocumentationLinkTests.cs:113	3.2 s	a DecompilerTextView exists right after boot; no decompile needed
Bookmarks/BookmarkGutterTests.cs (4) + BookmarkContextMenuTests.cs (3)	7.7 + ~4 s	Object/String as "another document"
Analyzers/Library/FieldAccessAnalyzerTests.cs:60 (7)	5.3 s	a new AnalyzerContext per Analyze() call (10 calls) over the whole 1.9 MB ILSpy.Tests.dll because the sample is internal; one shared context in OneTimeSetUp is pure data and allowed
AssemblyList/AssemblyTreeTests.cs selection tests (:1721, :919, :1093, :1674, :867, :1022), DecompileInNewViewTests.cs:211-384	~12 s	land on the CoreLib assembly node (default list order CoreLib, Uri, Linq), which rebuilds CoreLib's type system (~1 s) per test; they only need "several top-level rows"
PreviewTabPromotionTests.cs (6 tests with Uri as the second type), DecompilerViewTests.cs:105	~5 s	~1 s teardown cancel per Uri decompile; Version -> Greeter

B. Dead fixed waits (~5 s, nine one-line deletions)

await Task.Delay(600) at Navigation/NavigationTests.cs:64,114,117,179,
Navigation/BrowseBackForwardCommandTests.cs:113,282 (helper used by 5 tests),
Commands/HelpCommandTests.cs:213, AssemblyList/AssemblyListResetTests.cs:95,
AssemblyList/SwitchListAndDeleteAllTests.cs:56 wait out "NavigationHistory collapses within
0.5 s". ILSpy/NavigationHistory.cs:124 only collapses a repeat of the same entry; every site
records a different one. BrowseBackForwardCommandTests.cs:246 builds a history it never uses.

C. Strictly subsumed tests (same production path, assertions a subset) (~5 s)

• ApiVisibilityFilterTests.cs:87 in :106
• AssemblyTreeTests.cs:1340 in :1390; :1444 in :1468; :1497 in :1517
• PreviewTabPromotionTests.cs:59 in :430/:620; :45 in :620
• DecompileInNewViewTests.cs:54 in :75
• CommandLineArgumentsTests.cs:143 in :273 (move the TypeTreeNode type check there)
• BrowseBackForwardCommandTests.cs:51 in :289 (keep the MenuCategory assert)
• AnalyzeContextMenuTests.cs:178 vs AnalyzerTreeKeyboardTests.cs:146: both go through the
  single window KeyBinding to AnalyzeCommand.Execute; keep one
• AnalyzeContextMenuTests.cs:51 (TestHarness.GetEntry already asserts the header); :133 in :283
• AnalyzerNavigationTests.cs:48 in AnalyzerTreeKeyboardTests.cs:75 (borderline: direct call vs Enter)
• Near-duplicates with a small gap: AssemblyTreeTests.cs:1674 vs :1721, :1022 vs :1053,
  PreviewTabPromotionTests.cs:78 vs :121

D. Same setup and action, different assertions: mergeable (~15 s)

DecompilerViewTests.cs:346+375, :105+477; PreviewTabPromotionTests.cs:121+671,
:214/:288/:506/:544; DecompileInNewViewTests.cs:340+384; AssemblyTreeTests.cs:59+179,
:207+261, :489+537; AssemblyTree/AssemblyTreeModelTests.cs:147/:170/:193 (its
CreatePackage() also emits 2 DLLs + a zip per test, 5x); NavigationTests.cs:38 +
BrowseBackForwardCommandTests.cs:200; BookmarkGutterTests.cs:48/:94/:133;
ReferenceClickTests.cs:105+163; AnalyzeContextMenuTests.cs:61+87.

Kept as distinct (different entry points): menu vs keyboard vs mouse-button variants, Tab vs
Shift+Tab, middle vs Ctrl+middle click, pane method vs context menu.

E. Other

• Test counts: the three baselines ran the pre-existing 12:35 binaries, which carry one extra
  test (UpdatePanelViewModelTests.Automatic_update_checks_are_off_in_the_test_settings, not in
  HEAD); the experiment runs were rebuilt from HEAD and have 1351. The 0.02 s difference is
  irrelevant to the numbers above.
• MemberImplementsInterfaceAnalyzerTests (116 plain tests, 3.6 s) is fine; plain tests are not
  where the time is.

3. Totals and recommended order

lever	kind	wall-clock effect
A: fixture instead of framework types in ~45 tests	test edits, one fixture per commit	~75 s (20 %) measured upper bound from per-test times
B: delete 9 dead delays	trivial	~5 s
C + D: subsumed and mergeable tests	test edits	~20 s
harness knobs (poll interval, container skip)	measured	not distinguishable from noise; skip
2 shards on CI (Linux/macOS step)	workflow edit	269 s -> ~165 s per job; Windows unchanged
4 shards locally	script	369 s -> ~120-140 s

Doing A-D first takes the single-process suite from ~369 s to ~270 s and makes the shards
cheaper too (less CoreLib churn, which is what the concurrent runs contend on). Order by seconds
per edit: AnalyzerResultHighlightTests, ReferenceClickTests setup, CommandLineArgumentsTests:84,
BookmarkAnchoringTests, SetActiveDockableTests, BookmarkNavigationViewTests,
TabFocusNavigationTests, ILLanguageTooltipTests, FieldAccessAnalyzerTests, then B, then C/D,
then the CI step.

---

Assisted-by: Claude:claude-fable-5-1:Claude Code