CI: tier the test matrix into PR / main / nightly to cut Actions minutes

Aperta
  • Qualcosa non va?
#432 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 5 giorni

@napetrov ci sta già lavorando.

Dal 23/10/2026.

  • #451 di @napetrov — aperta
  • #452 di @napetrov — aperta

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
32/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cmake, cpp, github-actions
Ambito
ci-cd, devops

Direzione di ricerca

Start by reading the GitHub Actions workflow definitions and the referenced test and CMake configuration files, including tests/CMakeLists.txt and cmake/options.cmake. Compare each matrix entry with the coverage policy proposed in the issue, then agree on which jobs run for PRs, pushes to main, and nightly schedules. Done means the agreed tiers run the intended coverage and the per-push cost is reduced; the issue leaves that policy open for discussion.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

The Intel OSPO flagged ScalableVectorSearch as using ~80k GitHub Actions minutes per month. Measured over 2026-09-16..10-16: 79.7k runner-minutes across 1,158 workflow runs. About 90% of that comes from pull_request events: each push to a PR runs ~25 jobs, and PRs average ~5 pushes.

This issue proposes splitting CI into tiers. Coverage policy is a team decision, so it is up for discussion here before any PR.

Where the minutes go

Workflow Minutes / 30d Per green push
build-linux 36.5k 253 min (7 jobs)
build-linux-arm 19.1k 113 min (3 jobs)
build-cpp-runtime-bindings 10.8k 68 min
build-macos 8.2k 56 min
build-c-api-bindings 3.3k 22 min
cibuildwheel 1.6k 9 min

The cost of a push went from 346 to 521 runner-minutes after #369 (2026-10-01).

What each matrix entry uniquely covers

  • ivf=OFF is a strict subset of ivf=ON. IVF only adds sources (tests/CMakeLists.txt IVF lists, utils, benchmark) and the MKL link. Nothing in include/ checks SVS_ENABLE_IVF. The build-without-MKL path is already covered by the clang-18 ASAN job, which is ivf=OFF. That makes the three RelWithDebInfo ivf=OFF jobs redundant.

  • Compilers:

    • There is almost no compiler-specific code (meta.h:260, preprocessor.h:58/64).
    • g++-11 is the release toolchain; the LTO archive requires GCC 11.2.
    • clang-15 is the oldest supported clang.
    • clang-18 is ASAN.
    • macOS clang-20 is libc++.
    • g++-12 adds little, and builds with -Wno-uninitialized (cmake/options.cmake).

  • ARM has no NEON code, so it runs the generic kernels. Its unique value is the weak memory model for the concurrent index, and one compiler is enough to cover that. Today it runs three.

  • Not covered by any job today:

    • the TSan targets (SVS_EXPERIMENTAL_ENABLE_CONCURRENT_TSAN)
    • integration tests under ASAN (they are not compiled in Debug)
    • SVS_NO_AVX512=YES
    • NUMA
    • the full-size concurrent stress tests, after #433

    AVX-512 coverage depends on which runner a job lands on: 16 of the 26 sampled runners lacked AVX-512, and 14 of those were AMD EPYC 7763.

Proposal

Tier Jobs ≈ min per push
PR x86 g++-11 ivf=ON (+ binaries and examples), x86 clang-18 ASAN ivf=OFF, ARM g++-12, cpp-runtime ×2, c-api ×2 + integration, cibuildwheel ~150–180
push to main PR tier + clang-15 ivf=ON, g++-12 ivf=ON, macOS clang-20, FAISS integration ~375
nightly (schedule) main tier + ARM g++-11 / clang-15, SVS_TEST_CONCURRENT_FULL_SIZE=ON, TSan targets, SVS_FORCE_INTEGRATION_TESTS under ASAN, SVS_NO_AVX512=YES, FAISS main HEAD n/a

A failure that only the main or nightly tier can catch shows up after merge rather than before. Two ways to get that coverage before merge when it matters:

  • a full-ci label that runs the main-push tier on a PR;
  • running macOS automatically when a PR touches platform-sensitive paths (allocator, datatype.h).

The per-push minute figures are estimates computed from median job durations, not measurements.

Related

  • #433: concurrent tests at small size by default. Measured: those tests take 107 s instead of ~550 s on 4 cores.
  • #434: three flaky tests that cost ~2.8k min/month in failed jobs and reruns.
  • #435: paths-ignore filters. Replayed on the last 30 days of PRs, they skip ~12k PR minutes.
  • Not in this issue, for separate follow-ups:
    • prebuilt Docker image in GHCR (the image is rebuilt 8–9 times per push at 3.8 min each);
    • pinning FAISS and caching its build;
    • the static c-api variant failing on main since 10-09 because of skew against the nightly LTO archive (see #387);
    • no required status checks are configured on main.
Lingua principale
C++
Stelle
241
Fork
48
Merge medio
4g 21h
PR unite (30g)
11

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di intel/ScalableVectorSearch

Tutte le issue di intel/ScalableVectorSearch

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.