CI: tier the test matrix into PR / main / nightly to cut Actions minutes

Offen
  • Stimmt etwas nicht?
#432 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 5 Tagen

@napetrov arbeitet bereits daran.

Seit 23.10.2026.

  • #451 von @napetrov — offen
  • #452 von @napetrov — offen

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
32/100
Issue-Typ
Refactoring
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
cmake, cpp, github-actions
Bereich
ci-cd, devops

Rechercherichtung

Start by reading the GitHub Actions workflow definitions and the referenced test and CMake configuration files, including tests/CMakeLists.txt and cmake/options.cmake. Compare each matrix entry with the coverage policy proposed in the issue, then agree on which jobs run for PRs, pushes to main, and nightly schedules. Done means the agreed tiers run the intended coverage and the per-push cost is reduced; the issue leaves that policy open for discussion.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

The Intel OSPO flagged ScalableVectorSearch as using ~80k GitHub Actions minutes per month. Measured over 2026-09-16..10-16: 79.7k runner-minutes across 1,158 workflow runs. About 90% of that comes from pull_request events: each push to a PR runs ~25 jobs, and PRs average ~5 pushes.

This issue proposes splitting CI into tiers. Coverage policy is a team decision, so it is up for discussion here before any PR.

Where the minutes go

Workflow Minutes / 30d Per green push
build-linux 36.5k 253 min (7 jobs)
build-linux-arm 19.1k 113 min (3 jobs)
build-cpp-runtime-bindings 10.8k 68 min
build-macos 8.2k 56 min
build-c-api-bindings 3.3k 22 min
cibuildwheel 1.6k 9 min

The cost of a push went from 346 to 521 runner-minutes after #369 (2026-10-01).

What each matrix entry uniquely covers

  • ivf=OFF is a strict subset of ivf=ON. IVF only adds sources (tests/CMakeLists.txt IVF lists, utils, benchmark) and the MKL link. Nothing in include/ checks SVS_ENABLE_IVF. The build-without-MKL path is already covered by the clang-18 ASAN job, which is ivf=OFF. That makes the three RelWithDebInfo ivf=OFF jobs redundant.

  • Compilers:

    • There is almost no compiler-specific code (meta.h:260, preprocessor.h:58/64).
    • g++-11 is the release toolchain; the LTO archive requires GCC 11.2.
    • clang-15 is the oldest supported clang.
    • clang-18 is ASAN.
    • macOS clang-20 is libc++.
    • g++-12 adds little, and builds with -Wno-uninitialized (cmake/options.cmake).

  • ARM has no NEON code, so it runs the generic kernels. Its unique value is the weak memory model for the concurrent index, and one compiler is enough to cover that. Today it runs three.

  • Not covered by any job today:

    • the TSan targets (SVS_EXPERIMENTAL_ENABLE_CONCURRENT_TSAN)
    • integration tests under ASAN (they are not compiled in Debug)
    • SVS_NO_AVX512=YES
    • NUMA
    • the full-size concurrent stress tests, after #433

    AVX-512 coverage depends on which runner a job lands on: 16 of the 26 sampled runners lacked AVX-512, and 14 of those were AMD EPYC 7763.

Proposal

Tier Jobs ≈ min per push
PR x86 g++-11 ivf=ON (+ binaries and examples), x86 clang-18 ASAN ivf=OFF, ARM g++-12, cpp-runtime ×2, c-api ×2 + integration, cibuildwheel ~150–180
push to main PR tier + clang-15 ivf=ON, g++-12 ivf=ON, macOS clang-20, FAISS integration ~375
nightly (schedule) main tier + ARM g++-11 / clang-15, SVS_TEST_CONCURRENT_FULL_SIZE=ON, TSan targets, SVS_FORCE_INTEGRATION_TESTS under ASAN, SVS_NO_AVX512=YES, FAISS main HEAD n/a

A failure that only the main or nightly tier can catch shows up after merge rather than before. Two ways to get that coverage before merge when it matters:

  • a full-ci label that runs the main-push tier on a PR;
  • running macOS automatically when a PR touches platform-sensitive paths (allocator, datatype.h).

The per-push minute figures are estimates computed from median job durations, not measurements.

Related

  • #433: concurrent tests at small size by default. Measured: those tests take 107 s instead of ~550 s on 4 cores.
  • #434: three flaky tests that cost ~2.8k min/month in failed jobs and reruns.
  • #435: paths-ignore filters. Replayed on the last 30 days of PRs, they skip ~12k PR minutes.
  • Not in this issue, for separate follow-ups:
    • prebuilt Docker image in GHCR (the image is rebuilt 8–9 times per push at 3.8 min each);
    • pinning FAISS and caching its build;
    • the static c-api variant failing on main since 10-09 because of skew against the nightly LTO archive (see #387);
    • no required status checks are configured on main.
Vorherrschende Sprache
C++
Sterne
241
Forks
48
Ø Merge
4 T. 21 Std.
Gemergte PRs (30 T.)
11

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus intel/ScalableVectorSearch

Alle Issues in intel/ScalableVectorSearch

Ähnliche Issues

Weitere Issues zu C++

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.