Make CI stable and faster

Aperta
  • Qualcosa non va?
#2,913 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 5 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
20/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
csharp, github-actions, unity
Ambito
ci-cd, testing

Direzione di ricerca

Start by reading the listed PRs in their stated merge order, beginning with getsentry/app-runner#65, then follow the sentry-unity stack from #2911 down to #2907. The issue describes multiple CI flakes, retries, cache changes, and timing fixes, with results from reruns and a full matrix run; done means the stack is merged bottom-up and CI remains stable and faster.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Why

CI on main is too flaky and too slow to trust:

  • Only 29–30% of PR runs (33–40% on main) are green on the first attempt. Most red runs pass on an unchanged rerun.
  • A green PR run takes 41 min at p50 and 80 min at p90.
  • Unity license retries cost about 9 min per build job, 7,150 min over 30 days. A PR run starts 14 build jobs against 7–8 floating licenses shared with the team.
  • Caches sit at ~11 GB against the 10 GB quota, so LRU eviction keeps runs cold.

A red run that goes green on a rerun teaches everyone to ignore red, and real regressions hide in the noise.

What

A stack of small PRs, each fixing one cause:

  • Fail fast: stop the run when the SDK build fails, and cap job run time with timeout-minutes.
  • Retries: retry known infra failures once, by a specific log signature that is printed when it fires. Examples: LicensingClient IPC timeout, an editor killed at startup, a Unity shutdown crash, Gradle 429, a segfaulting build tool, sentry-cli 5xx, an ingest 5xx on the crash report. Product and test bugs get fixed, not retried.
  • Root-caused flakes:
    • Native symbolication: sentry-cli returns before sentry.io has processed the upload, so events arrive before their symbols. The upload assertion now waits until sentry.io lists the uploaded debug files.
    • iOS first launch: a launch less than ~2 min after the simulator boots can stall for minutes. The simulator now boots while the app compiles.
    • Unity 6000.6 in Docker: Unity kills the PID in a Library/burst.pid copied from the Create job, which turned out to be the editor's own xvfb-run. The stale pid files are now deleted before each build.
    • An Android test app that never initialises is relaunched once.
    • App-hang tracking is on only in the App Hang test, so its 2 s watchdog can't add a second event to the crash test on a slow simulator.
    • The playmode integration tests get 5 s to receive an event.
    • ConfigureOptions run a second time cleared the options configuration. This is an SDK bug users can hit too, with a CHANGELOG entry and EditMode tests.
  • Speed:
    • Two fewer Unity launches per integration test build, 52 → 24 launches per PR run.
    • Integration test and native SDK caches keyed on their inputs.
    • The leftover .NET 9 install and the workload restore are gone.
    • The iOS size baseline and the simulator terminate stall are off the Run iOS critical path.
    • The license retry is jittered and capped at 30 min.

Results so far

Ten back-to-back reruns of the unchanged stack tip:

Before (main 626d790c) Stack tip
First-attempt green, PR 29–30% 8/10
PR wall time p50/p90 41/80 min 32.0/35.2 min
Unity launches per PR run 52 24
License-retry sleep ~9 min per build job 217–1,029 s per run in total
  • One of the two red runs was the ConfigureOptions bug, fixed since in #2911; its final head is green on the first attempt in 27.3 min.
  • The other was a one-off Unity-internal Bee IPC error (Request cancelled while reading the binlog).
  • A full-matrix run (5 Unity versions) of the stack is green.

Merge order

The PRs are stacked; each one is based on the one below it. Merge from the bottom up. sentry-unity squash-merges, so after each merge the next PR is rebased onto main and its CI reruns.

app-runner goes first. The stack pins its PRs' exact commits, and merging them with merge commits keeps those pins valid:

  1. getsentry/app-runner#65 (pinned by #2910)
  2. getsentry/app-runner#64 (pinned by #2901)
  3. main (app-runner)

sentry-unity, from the tip of the stack down to main:

  1. #2911 Fix the symbolication, iOS first-launch, ingest 5xx and burst.pid flakes, plus the ConfigureOptions fix
  2. #2910 Take the iOS size baseline and the terminate stall off the Run iOS path (needs app-runner#65)
  3. #2909 Key the native SDK caches on their own inputs and skip the workload restore
  4. #2908 Recover from LicensingClient IPC timeouts, sentry-cli 5xx and flush timeouts
  5. #2906 Give the playmode integration tests 5 s to receive an event
  6. #2905 Cap job run time with timeout-minutes
  7. #2904 Retry known infra failures once, by signature
  8. #2903 Launch Unity two fewer times per integration test build
  9. #2902 Key the integration test caches on their inputs
  10. #2901 Relaunch an Android test app that never initialised once (needs app-runner#64)
  11. #2900 Jitter the Unity license retry and cap it at 30 minutes
  12. #2899 Enable app-hang tracking only in the App Hang integration test
  13. #2898 Fail fast when the SDK build fails
  14. #2907 Drop the leftover .NET 9 install from the Linux native SDK build
  15. main
Lingua principale
C#
Stelle
295
Fork
63
Merge medio
4g 10h
PR unite (30g)
29

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di getsentry/sentry-unity

Tutte le issue di getsentry/sentry-unity

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.