C# build targets duplicate the Copilot CLI bundle per source path: no shared cache, archive kept, payload copied twice

Aperta
  • Qualcosa non va?
#2,903 0 commenti 1 reazione 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
backend, build-system

Direzione di ricerca

Start with build/GitHub.Copilot.SDK.targets in the NuGet package; inspect how _CopilotCacheDir, the archive download and extraction, and native payload staging are defined. Reproduce with the two source paths in the issue and compare archive and output contents. Done when repeated builds avoid the reported duplicate storage while still providing the required native files to each app.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

The GitHub.Copilot.SDK MSBuild targets cache the downloaded Copilot CLI bundle under
$(IntermediateOutputPath)copilot-cli/<version>/<platform>/. Because the intermediate output path of a
.NET 10 file-based app (dotnet run app.cs) is a per-source-path directory in the user's runfile cache,
every distinct source path re-downloads and re-extracts the CLI bundle. Nothing is shared between
entries, so identical bytes are duplicated once per path.

On a machine where a test harness ran the same app from many temporary directories, this produced
~1,400 copies of the same 84 MB archive and its extracted payload — 755 GB.

This is the SDK-package half of a larger report; the file-based-app cache-keying half is filed at
dotnet/sdk#56940.

Environment

GitHub.Copilot.SDK 1.0.26   (CopilotCliVersion 1.0.103)
.NET SDK 10.0.402
macOS 26 (arm64), APFS

~/.nuget/packages/github.copilot.sdk/1.0.26/build/GitHub.Copilot.SDK.targets:

<_CopilotCacheDir>$(IntermediateOutputPath)copilot-cli\$(CopilotCliVersion)\$(_CopilotPlatform)</_CopilotCacheDir>
<_CopilotArchivePath>$(_CopilotCacheDir)\copilot.tgz</_CopilotArchivePath>
...
<DownloadFile SourceUrl="$(_CopilotChecksumsUrl)" DestinationFolder="$(_CopilotCacheDir)" ... />
<DownloadFile SourceUrl="$(_CopilotDownloadUrl)" DestinationFolder="$(_CopilotCacheDir)" DestinationFileName="copilot.tgz" ... />
<Exec Command="&quot;$(_TarCommand)&quot; -xzf &quot;$(_CopilotArchivePath)&quot; --strip-components=1 -C &quot;$(_CopilotCacheDir)&quot;" ... />

The cache is scoped to IntermediateOutputPath, which is scoped to the project — and for a file-based
app the project is per source path.

Reproduction

// repro.cs
#:package [email protected]
Console.WriteLine("repro");

CACHE="$HOME/Library/Application Support/dotnet/runfile"

mkdir -p /tmp/rfr/a /tmp/rfr/b
cp repro.cs /tmp/rfr/a/ && cp repro.cs /tmp/rfr/b/   # byte-identical

(cd /tmp/rfr/a && dotnet run repro.cs)   # cold: 4.4s, creates repro-<hash A>
(cd /tmp/rfr/a && dotnet run repro.cs)   # warm: 0.14s, no new directory
(cd /tmp/rfr/b && dotnet run repro.cs)   # cold again: 4.7s, creates repro-<hash B>

Both cache directories are 546 MB. The archive is byte-identical in both, and physically duplicated:

$ stat -f 'inode=%i links=%l size=%z' \
    $CACHE/repro-552f7c…/obj/debug/copilot-cli/1.0.103/darwin-arm64/copilot.tgz \
    $CACHE/repro-16b020…/obj/debug/copilot-cli/1.0.103/darwin-arm64/copilot.tgz
inode=96355505 links=1 size=83944573 …/repro-552f7c…/copilot.tgz
inode=96356143 links=1 size=83944573 …/repro-16b020…/copilot.tgz

$ shasum -a 256 …/repro-552f7c…/copilot.tgz …/repro-16b020…/copilot.tgz
4cba81686f2d4a56e2ea2e1bd1cdde303d07f28de11593e1db1786f1219b8f74  …/repro-552f7c…/copilot.tgz
4cba81686f2d4a56e2ea2e1bd1cdde303d07f28de11593e1db1786f1219b8f74  …/repro-16b020…/copilot.tgz

Per-artifact cost breakdown (one cold build, 546 MB)

284 MB  obj/debug/copilot-cli/1.0.103/darwin-arm64/
         81 MB   copilot.tgz                       (the downloaded archive, kept after extraction)
         98 MB   prebuilds/darwin-arm64/runtime.node
         15 MB   foundry-local-sdk/…/libfoundry_local.dylib
         12 MB   prebuilds/darwin-arm64/libOneAuthInterop.dylib
261 MB  bin/debug/runtimes/osx-arm64/native/
         98 MB   runtime.node
         98 MB   libcopilot_runtime.dylib
         12 MB   libOneAuthInterop.dylib
 18 MB  bin/debug/GitHub.Copilot.SDK.dll
0.3 MB  the app's own dll + pdb

Two further observations from the same measurement:

  1. bin/debug/runtimes/osx-arm64/native/runtime.node and libcopilot_runtime.dylib are the same
    file.     Both are 98 MB with sha256 70280452e26d03c3b6bd77b47510460b70416e0e421eac336c134a397a964fc8
    and links=1 — the same payload is copied twice into one output directory. The targets comment says
    the cdylib is "renamed from the tarball's napi-style prebuilds//runtime.node", so this looks
    like both the original name and the renamed copy surviving.
  2. copilot.tgz is kept after extraction (81 MB). The extraction target never deletes it, so the
    archive is retained in every cache directory alongside the extracted tree it produced.

So even a single, correctly reused artifact directory holds the payload roughly three times over
(obj/…/prebuilds/runtime.node, bin/…/runtime.node, bin/…/libcopilot_runtime.dylib), plus the
archive.

Workaround

Setting ArtifactsPath keeps the build output out of the shared cache:

dotnet run repro.cs --artifacts-path /tmp/scratch

This does work — build output lands under /tmp/scratch — but the runfile cache still gains an ~84 KB
stub directory per source path (csc.rsp, build-start.cache, build-success.cache), so it is a
mitigation rather than a fix.

Possible directions

  1. Cache the download under a location keyed by CopilotCliVersion + platform only (e.g. under
    $(NuGetPackageRoot) or a per-user CLI cache), and have each project's intermediate output link or
    copy from there. The archive and extracted payload are identical for every consumer of the same
    version.
  2. Delete copilot.tgz after successful extraction.
  3. Avoid emitting the same 98 MB payload under both runtime.node and libcopilot_runtime.dylib in the
    same output directory.
  4. Prefer hard links (or a copy-if-newer that preserves them) when staging the bundle into the output
    directory.

Impact

A five-line app referencing this package costs 546 MB of build output per distinct source path. That is
survivable for a normal project, but a test harness, CI matrix, or agent sandbox that generates a fresh
directory per case turns it into unbounded disk growth: ~1,400 directories, 755 GB on the machine this
was measured on.

Lingua principale
Java
Stelle
10.7k
Fork
1.5k
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di github/copilot-sdk

Tutte le issue di github/copilot-sdk

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.