[EPIC] Improve end-to-end C# SDK generation performance across all five stages

Start with the reproduction command and the linked eng/CodeGeneration.targets to understand the existing generation workflow; compare the measured stages for the large and small workloads. The issue lists possible improvements across multiple owners but does not select a specific change, so clarify scope before starting. Done should include measured end-to-end results on representative workloads and unchanged generated output.

Improve end-to-end C# SDK generation performance across all five stages

Summary

We profiled end-to-end TypeSpec-based Azure management SDK generation on Windows, using a large SDK (Azure.ResourceManager.Network) and a small SDK (Azure.ResourceManager.Billing.Trust).

The bottleneck depends strongly on workload:

• Network: 791.744 seconds (13m12s). The C# generator dominates, particularly Roslyn semantic simplification.
• Billing Trust: 136.407 seconds (2m16s). Repeated npm installation and Git synchronization dominate; C# generation is only about 18% of the total.

This issue summarizes measured results and potential improvements across all five stages. Some orchestration changes belong in Azure SDK tooling rather than this repository; please treat those as coordinated follow-up work, not necessarily changes to the TypeSpec compiler.

No particular historical regression commit has been established.

Reproduction and environment

From either package directory in Azure/azure-sdk-for-net:

dotnet build /t:GenerateCode -tl:off /p:SaveInputs=true

SDK repository revision: 8cb38f5a1bae4659bd18efea839e2e6c26fee10d.

Measured toolchain:

• Windows, 16 logical processors, approximately 64 GiB RAM.
• Network: .NET SDK 10.0.402, generator runtime .NET 10.0.13, Node 26.3.1.
• Billing Trust: .NET SDK 10.0.403, generator runtime .NET 10.0.14, Node 26.12.1.
• TypeSpec client generator CLI: 0.33.1.
• @azure-typespec/http-client-csharp-mgmt: 1.0.0-alpha.20261019.1.
• @azure-typespec/http-client-csharp: 1.0.0-alpha.20261014.4.
• @typespec/http-client-csharp: 1.0.0-alpha.20261014.5.
• Loaded Roslyn assemblies: 4.8.0.0, except in the isolated upgrade experiment below.

The Network sync follow-up used specification commit 9d0894a7bb035723be12f4b20a5d0633a434626c; Billing Trust used e9ab1308aa1e141278c625990cd49f77200b1a69. The input sizes and package versions above identify the profiled workloads; timings are not measurements of today's latest packages.

Five-stage overview

Stage	Network	Billing Trust
1. MSBuild startup/restore and tsp-client installation	~39 s	38.821 s
2. CLI startup, specification sync/copy, entrypoint discovery	~178 s	35.753 s
3. Emitter dependency installation	~28 s	29.571 s
4. TypeSpec import, compilation, JavaScript emission	~30 s	4.160 s
5. .NET C# generation and post-processing	~516 s	24.637 s
Other coordination/finalization	Included in approximate boundaries	3.466 s
Measured end-to-end total	791.744 s	136.407 s

Network boundaries are approximate, derived from MSBuild logs and a process timeline sampled about every 2.3 seconds. Billing Trust boundaries were measured with subprocess/filesystem instrumentation; the residual is reported separately. Rounded rows need not sum exactly.

Stage 1: MSBuild startup/restore and tsp-client installation

Findings

Billing Trust spent 10.530 s in MSBuild startup/restore and 28.291 s in npm ci for tsp-client. The installation contained 130 packages.

The original installation had 130 tarball cache hits and zero tarball downloads, with no dependency lifecycle scripts executed. Warm tarball caches therefore do not eliminate installation overhead: npm ci still recreates node_modules.

The installed CLI tree contained 6,672 files / 34,439,677 logical bytes. Isolated follow-up controls took 10.051 s and 10.042 s, illustrating substantial run/cache variability; that difference must not be credited to a new optimization.

The SDK MSBuild target currently invokes CLI npm ci unless SkipTspClientInstall is set:
CodeGeneration.targets.

Potential improvements

• Reuse a verified installed CLI tree, keyed by manifests, lockfile, Node/npm versions, OS/architecture, and relevant installation options.
• Validate required entrypoints and successful-install state rather than checking only that node_modules exists.
• Publish shared caches atomically with per-key locking; retain clean installation on misses and where policy requires it.
• Keep install/audit policy intact; do not blindly replace npm ci with npm install.
• Document the existing /p:SkipTspClientInstall=true local-iteration control and its invalidation requirements.
• Measure MSBuild startup/restore separately before proposing further changes there.

Likely owner: Azure SDK MSBuild/orchestration tooling.

Stage 2: Specification synchronization, copying, and discovery

Findings

Network's selected specification directory contained 36,780 files, including 36,595 JSON files and only 167 TypeSpec files. Its historical stable subtree alone contained 32,429 files. The copied directory occupied approximately 182 MB of logical contents.

The CLI sparsely checks out the selected service directory, recursively copies it, removes the temporary checkout, and recursively searches for the entrypoint. This carries a large historical Swagger tree through a TypeSpec workflow.

Billing Trust's corresponding tree had only 35 files / 155,365 bytes:

Billing Trust synchronization operation	Duration
CLI startup before clone	3.505 s
git clone --no-checkout --filter=tree:0	18.327 s
Sparse-checkout init + add	0.290 s
Checkout pinned commit	13.289 s
Copy selected specification tree	0.097 s
Delete temporary checkout	0.026 s
Copy emitter manifests	0.021 s
Recursive entrypoint discovery	0.002 s
Other coordination	0.196 s

Thus copying is material for the large workload, but negligible for the small one.

Separate Network pre-copy timing runs measured:

Operation	First run	Repeat
CLI startup/pre-clone preparation	21.173 s	3.436 s
Clone	11.810 s	10.286 s
Sparse-checkout init + add	0.190 s	0.264 s
Checkout	48.312 s	54.158 s
Total through pre-copy boundary, including small gaps	81.489 s	68.149 s

Git Trace2 showed that checkout includes deferred network retrieval, not just local file creation. In the Network repeat, three sequential fetches consumed 22.626 s of the 54.158-second checkout. Billing Trust similarly spent 12.050 s of its 13.289-second checkout in three fetches. These fetch times are subdivisions of checkout, not additional pipeline stages.

The Network pre-copy runs intentionally stopped before copying; they are new measurements, not an exact reconstruction of the original ~178-second stage.

Potential improvements

• Reuse a commit-addressed specification/object cache instead of creating a fresh clone for every generation; isolate working directories and validate the pinned commit.
• Benchmark partial-clone/filter strategies and batched object retrieval against real large and small service trees. Do not equate a fast initial clone with completed downloading.
• Copy only the dependency closure needed for compilation, including imported TypeSpec, referenced examples, and configuration. Do not indiscriminately exclude all JSON.
• Resolve an explicitly configured or root client.tsp before falling back to recursive discovery.
• Avoid duplicate checkout/copy trees when a safe immutable cached source can be consumed directly.
• Emit separate timings for clone, deferred fetch, materialization, copying, discovery, and cleanup.

Likely owner: TypeSpec client-generation CLI/specification orchestration.

Stage 3: Emitter dependency installation

Findings

Billing Trust's emitter npm ci took 29.571 s for 153 packages. All 153 tarballs were cache hits, with no lifecycle scripts executed.

The installed emitter tree contained 18,152 files / 284,186,588 logical bytes. Follow-up controls took 25.683 s and 25.364 s; npm's unpack intervals were approximately 20-24 s. Re-extraction, rather than downloading tarballs, is the main observed cost.

Audit work overlaps extraction. In every isolated trial, the audit interval was shorter than the unpack interval; these timings must not be added. --prefer-offline reduced some metadata/audit timing but did not demonstrate a consistent large wall-time improvement.

Three emitter distributions also vendored .NET generator payloads:

Distribution	Generator payload logical bytes
Azure management C#	74,920,705
Azure C#	73,137,465
Unbranded C#	32,950,666

Hash comparison found 125 distinct contents repeated across these distributions, totaling 137,912,757 redundant logical bytes relative to keeping one copy of each content. This is potential packaging savings, not a measured install-time reduction or proof that these files can simply be deleted.

Potential improvements

• Cache complete installed emitter dependency trees with the same correctness/invalidation rules as Stage 1.
• Keep reusable trees outside temporary per-generation directories deleted by the normal workflow.
• Expose a safe orchestration path for reuse; SkipSync=true currently does not skip emitter installation.
• Investigate sharing the .NET runtime/generator payload across layered packages. Validate assembly loading, plugins, resources, and version compatibility.
• Do not expect npm dedupe to deduplicate files vendored inside different npm packages.
• Preserve audits and lifecycle behavior; neither disabling audit nor changing antivirus settings was established as an appropriate fix.

Existing supported iteration experiment: after validating installed versions against both locks and preserving the specification workspace, tsp-client generate --skip-install completed in 55.883 s and 27.690 s, with unchanged output. These runs also skipped Git sync and outer MSBuild, so their improvement over 136.407 s is not an installation-only A/B result.

Likely owners: client-generation CLI and layered C# emitter packaging.

Stage 4: TypeSpec compilation and JavaScript emission

Findings

This stage took approximately 30 s for Network and 4.160 s for Billing Trust. The interval includes TypeSpec import/setup and emission, not just the compiler's compile call.

Saved input model sizes:

• Network: 49,453,903 bytes; 1,133 models and 301 enums.
• Billing Trust: 544,991 bytes; 30 models, 13 enums, one top-level client.

This stage is not the dominant observed bottleneck in either workload. We did not isolate compiler import, checking, emitter traversal, and model serialization sufficiently to attribute their individual costs.

Potential improvements

• Add structured substage timings for module import/setup, program construction/checking, emitter traversal, and serialized model production.
• Establish small/large workload benchmarks before making compiler/emitter changes.
• Investigate reuse/incrementality for unchanged inputs only with dependency/configuration-aware invalidation.
• Use profiling evidence to select changes; the current measurements do not establish that compiler parsing or model serialization is the bottleneck.

Likely owners: TypeSpec compiler and C# emitter, guided by finer-grained measurements.

Stage 5: C# generator and Roslyn post-processing

Findings

Network spent approximately 516 s in the C# generator. Its 3,828 write events corresponded to 3,684 unique source files; 144 resource-serialization paths were written twice.

The primary bottleneck is whole-library Roslyn simplification:

• The generator annotates entire generated roots with Simplifier.Annotation.
• It schedules document processing for all generated documents using Task.WhenAll.
• Each document builds a semantic model and invokes Simplifier.ReduceAsync.

Source at the investigated revision:
GeneratedCodeWorkspace.cs, lines 90-160.

A late 90-second sampled-thread-time capture showed:

• CSharpNameReducer.SimplifyName: 60.78% inclusive sampled thread time.
• NameSimplifier.TrySimplify: 43.32% inclusive.
• Monitor.Enter_Slowpath: 26.92% exclusive.

Contention stacks led through syntax-annotation helpers and ConditionalWeakTable.Add. These are sampled thread-time percentages, including waits, not wall-time/CPU-utilization percentages; inclusive entries overlap.

Experiments

Disable Roslyn reduction, same saved Network inputs: 85.944 s versus approximately 516 s in the original full run. Document post-processing dropped to about 2.324 s. However, 3,679 tracked source files changed textually, primarily retaining fully qualified global:: names. This is diagnostic evidence, not an output-transparent production fix.

Limit perceived processor count to two, simplification still enabled: 481.295 s, including 397.583 s in document post-processing. Output remained identical. This did not remove the bottleneck and does not justify a blanket processor-count recommendation.

Isolated Roslyn 5.9.0 upgrade, simplification enabled:

Saved-input C# replay	Roslyn 4.8.0 control	Roslyn 5.9.0 A	Roslyn 5.9.0 B
Generator wall time	586.815 s	524.712 s	531.142 s
Roslyn document processing	406.469 s	347.391 s	355.846 s
Peak working set	3,721.512 MiB	3,434.691 MiB	3,435.922 MiB

The upgrade averaged 10.04% less generator wall time, 13.49% less document-processing time, and 7.69% lower peak working set.

Actual loaded Roslyn assembly versions were verified. Generator/plugin DLLs and inputs were unchanged; required runtime dependencies were updated in an isolated copy. All 3,684 unique generated source files were byte-identical across successful replays.

This was one matched control and two upgraded trials, not a statistically exhaustive benchmark. These are C#-only replay times and must not be mixed with the original full-pipeline run to manufacture a new end-to-end result. No production dependency pins were changed.

Billing Trust's C# generator took 24.637 s, including 8.286 s of Roslyn processing. Optimizing Roslyn alone therefore cannot address its setup-dominated total.

Potential improvements

• Evaluate a supported Roslyn upgrade with normal generator compatibility tests and performance coverage; the measured improvement is useful but insufficient alone.
• Investigate annotating only syntax that actually needs reduction, or generating appropriately simplified names earlier.
• Benchmark bounded document scheduling and semantic-model/cache reuse. Contention suggests a hypothesis, not proof that lowering parallelism will solve it.
• Investigate avoiding duplicated writes, preserving deterministic ordering and byte-equivalent final output.
• Measure startup/type-provider/visitor costs separately for small SDKs, where fixed process overhead matters more.
• Keep a fast local diagnostic mode available, while clearly distinguishing it from output-equivalent generation.
• NativeAOT was discussed but not benchmarked; no AOT performance benefit is established by this investigation.

Likely owners: upstream TypeSpec C# generator and Azure C# generator plugins.

Suggested acceptance criteria

• Structured machine-readable timings for all five stages, with non-overlapping boundaries and separate nested timings.
• Repeatable cold/warm benchmarks for both small and large SDKs, with toolchain, input revision, cache state, and loaded dependency versions recorded.
• Output-equivalence checks for optimizations intended to be transparent; compilation/API/serialization tests where behavior could change.
• Safe dependency/specification cache invalidation, interrupted-install recovery, concurrent-use handling, and clean-install fallback.
• No relaxation of dependency integrity, lockfile reproducibility, or audit policy.
• Separate follow-up work by owner, prioritizing installed-dependency/spec reuse for small SDKs and Roslyn reduction for large SDKs.

Measurement limitations

These are local workload studies, not a cross-machine performance guarantee. Network conditions and filesystem/package caches were not controlled; the two packages used different specification commits and minor runtime versions. Network's baseline had partial tracing instrumentation, while the later replay experiments did not.

SaveInputs=true preserves the final temporary specification/dependency workspace, so its normal end-of-run deletion is excluded; deletion of the separate sparse checkout is included. Generated outputs were checked for equality in the unchanged-output experiments. Detailed logs, timing summaries, hash manifests, and traces were retained locally but are not attached to this issue.