Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Full-suite test time: ReDoS/contract sweeps are 32% of the suite; gate them on PRs, add a slow-test budget, trim the top 10

Aperta
  • Qualcosa non va?
#5,166 6 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
10/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
github-actions, python
Ambito
ci-cd, testing-qa

Direzione di ricerca

Start with tests/tools/pr_gates.py for the existing ratchet pattern, smoke-test.yml for the full-suite parts, and the sweep tests under tests/extraction/. Parts A and B change CI gating and need maintainer design decisions first. Part C is the most separable: profile one slow test such as test_galaxyscope and keep its assertions. Done means the test passes with a shorter wall time and no weakened check.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

ci-cd github-actions performance scope:infra testing

Context (2026-10-14 CI audit, measured from origin/main .test_durations refreshed by #5151): the full suite (smoke-test.yml parts, required check full-suite) is the largest PR-path CI cost, 27.7% of runner time 21:00–24:00Z (squid-bot dashboard / #5164 table). Its time is concentrated:

Tests Serial time Share
Full suite (minus test_evidence_components.py, now Ground Truth's per #5163) 17,463 56.9 min 100%
Extraction ReDoS / contract sweeps (tests/extraction/**, *_redos_immune, *_redos_sweep, contract rules) 1,386 18.0 min 32%
Tests over 10 s 32 10.7 min 19%
Tests under 0.1 s 15,845 ~1.2 min 2%

Heaviest sweep functions (summed over params): test_ownership_contract_rules_are_redos_immune 3.3 min, test_high_risk_execution_contract_rules_are_redos_immune 3.0, globals 1.8, branch 1.6, import 1.5, state_mutation 0.9, safety 0.6, cleanup_contract_redos_sweep 0.6.

Already handled elsewhere (don't redo): evidence tests and the duplicate test_evidence_components run (#5162/#5163), part balancing and durations weight (#5133/#5162), Ground Truth (#5164).

Rule for every part: coverage never drops. Main pushes, nightly and dispatch keep running every test. Only when a test runs on a PR, or how fast, may change. When in doubt, run.

A. Run the extraction ReDoS/contract sweeps on PRs only when their inputs change
  • Mark the sweep tests (one pytest marker, e.g. redos_sweep) and define their input set: the extraction rule/regex files and contract specs they read, the sweep harness, and pyproject/requirements. In doubt, include the file.
  • On pull_request, the full-suite parts deselect the marker when the PR changes no input. The decision comes from the PR's changed files, computed the same way as Det Sweep's (#4825), and the required check still always reports. Push to main, nightly and dispatch run them always.
  • Tests for the classifier in both directions. PR body: the share of the last ~50 merged PRs that would have skipped, and the minutes saved.
B. Test-time ratchet: the suite must not get slower over time (owner, 2026-10-14)

Register it as a Ratchet in tests/tools/pr_gates.py (check command, update command, committed baseline), like the existing ratchets. CI runner timings vary ±20–30%, so it has two levels and only the robust one gates PRs:

  • Per-test gate (hard, every PR): a test whose CI duration exceeds 10 s fails unless it carries an explicit slow marker with a reason, or is listed in tests/test_time_baseline.json. Seed the baseline with today's 32 tests over 10 s and their times. A baseline test fails if it exceeds 2× its recorded time (wide enough for runner noise). The baseline only shrinks or gets faster: the update command lowers entries and drops ones under 10 s, and never raises one.
  • Whole-suite budget (hard on nightly main, report-only on PRs): total serial test seconds of the full suite, measured by the nightly run on main from its own junit/--durations (not the contended durations cache, see #5162). Nightly fails, and files or refreshes one issue, when the total exceeds the committed budget by >10%. When the suite gets faster, the update command lowers the budget to the new level (shrink-only). PRs print their suite total and the delta against the budget in the job summary, without gating.
  • Tests for the ratchet logic (threshold, 2× rule, shrink-only update, budget delta). Document it in .claude/rules/ (test-writing) and docs/ci.md.
C. Speed up the top 10 by hand (4.9 min, 8.6%), keeping the same assertions

Profile each and fix the cause (fixture scope, repeated scans, sampling size), or split it so -n auto can parallelise it. Don't weaken the check:

  • cultural_gauntlet::test_the_sampled_plan_passes_its_ratchet 60.8 s
  • test_unicode_gauntlet::test_the_oracle_is_sound_on_the_mainframe_seed 46.4 s
  • test_scan_path_independence_4058::test_the_same_tree_scans_the_same_under_any_parent[tmp] 42.2 s
  • test_cleanup_contract_2888::test_cleanup_contract_redos_sweep 33.2 s (or covered by A)
  • cobol_mainframe/test_tool_regex_redos::test_no_new_offenders 21.9 s
  • test_immutability_locks_contract_2772::test_immutability_locks_redos_sweep 21.0 s (or covered by A)
  • test_galaxyscope (3 tests: typosquat 19.5 s, z-score backfill 14.8 s, sarif purge 14.1 s): likely a shared expensive fixture
  • test_inline_extraction::test_a_spawned_pool_is_the_forked_pools_output 18.1 s

Acceptance: before/after full-suite part wall times on a typical ready PR (code-only and docs-only), and the runner-minutes per run, in each PR body. Test count on main and nightly unchanged.

Lingua principale
Python
Stelle
57
Fork
11
Merge medio
1h 32m
PR unite (30g)
990

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di squid-protocol/gitgalaxy

Tutte le issue di squid-protocol/gitgalaxy

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.