Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Full-suite test time: ReDoS/contract sweeps are 32% of the suite; gate them on PRs, add a slow-test budget, trim the top 10

Đã đóng
  • Có gì không ổn?
#5,166 6 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
10/100
Loại issue
Tái cấu trúc
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
github-actions, python
Lĩnh vực
ci-cd, testing-qa

Hướng nghiên cứu

Start with tests/tools/pr_gates.py for the existing ratchet pattern, smoke-test.yml for the full-suite parts, and the sweep tests under tests/extraction/. Parts A and B change CI gating and need maintainer design decisions first. Part C is the most separable: profile one slow test such as test_galaxyscope and keep its assertions. Done means the test passes with a shorter wall time and no weakened check.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

ci-cd github-actions performance scope:infra testing

Context (2026-10-14 CI audit, measured from origin/main .test_durations refreshed by #5151): the full suite (smoke-test.yml parts, required check full-suite) is the largest PR-path CI cost, 27.7% of runner time 21:00–24:00Z (squid-bot dashboard / #5164 table). Its time is concentrated:

Tests Serial time Share
Full suite (minus test_evidence_components.py, now Ground Truth's per #5163) 17,463 56.9 min 100%
Extraction ReDoS / contract sweeps (tests/extraction/**, *_redos_immune, *_redos_sweep, contract rules) 1,386 18.0 min 32%
Tests over 10 s 32 10.7 min 19%
Tests under 0.1 s 15,845 ~1.2 min 2%

Heaviest sweep functions (summed over params): test_ownership_contract_rules_are_redos_immune 3.3 min, test_high_risk_execution_contract_rules_are_redos_immune 3.0, globals 1.8, branch 1.6, import 1.5, state_mutation 0.9, safety 0.6, cleanup_contract_redos_sweep 0.6.

Already handled elsewhere (don't redo): evidence tests and the duplicate test_evidence_components run (#5162/#5163), part balancing and durations weight (#5133/#5162), Ground Truth (#5164).

Rule for every part: coverage never drops. Main pushes, nightly and dispatch keep running every test. Only when a test runs on a PR, or how fast, may change. When in doubt, run.

A. Run the extraction ReDoS/contract sweeps on PRs only when their inputs change
  • Mark the sweep tests (one pytest marker, e.g. redos_sweep) and define their input set: the extraction rule/regex files and contract specs they read, the sweep harness, and pyproject/requirements. In doubt, include the file.
  • On pull_request, the full-suite parts deselect the marker when the PR changes no input. The decision comes from the PR's changed files, computed the same way as Det Sweep's (#4825), and the required check still always reports. Push to main, nightly and dispatch run them always.
  • Tests for the classifier in both directions. PR body: the share of the last ~50 merged PRs that would have skipped, and the minutes saved.
B. Test-time ratchet: the suite must not get slower over time (owner, 2026-10-14)

Register it as a Ratchet in tests/tools/pr_gates.py (check command, update command, committed baseline), like the existing ratchets. CI runner timings vary ±20–30%, so it has two levels and only the robust one gates PRs:

  • Per-test gate (hard, every PR): a test whose CI duration exceeds 10 s fails unless it carries an explicit slow marker with a reason, or is listed in tests/test_time_baseline.json. Seed the baseline with today's 32 tests over 10 s and their times. A baseline test fails if it exceeds 2× its recorded time (wide enough for runner noise). The baseline only shrinks or gets faster: the update command lowers entries and drops ones under 10 s, and never raises one.
  • Whole-suite budget (hard on nightly main, report-only on PRs): total serial test seconds of the full suite, measured by the nightly run on main from its own junit/--durations (not the contended durations cache, see #5162). Nightly fails, and files or refreshes one issue, when the total exceeds the committed budget by >10%. When the suite gets faster, the update command lowers the budget to the new level (shrink-only). PRs print their suite total and the delta against the budget in the job summary, without gating.
  • Tests for the ratchet logic (threshold, 2× rule, shrink-only update, budget delta). Document it in .claude/rules/ (test-writing) and docs/ci.md.
C. Speed up the top 10 by hand (4.9 min, 8.6%), keeping the same assertions

Profile each and fix the cause (fixture scope, repeated scans, sampling size), or split it so -n auto can parallelise it. Don't weaken the check:

  • cultural_gauntlet::test_the_sampled_plan_passes_its_ratchet 60.8 s
  • test_unicode_gauntlet::test_the_oracle_is_sound_on_the_mainframe_seed 46.4 s
  • test_scan_path_independence_4058::test_the_same_tree_scans_the_same_under_any_parent[tmp] 42.2 s
  • test_cleanup_contract_2888::test_cleanup_contract_redos_sweep 33.2 s (or covered by A)
  • cobol_mainframe/test_tool_regex_redos::test_no_new_offenders 21.9 s
  • test_immutability_locks_contract_2772::test_immutability_locks_redos_sweep 21.0 s (or covered by A)
  • test_galaxyscope (3 tests: typosquat 19.5 s, z-score backfill 14.8 s, sarif purge 14.1 s): likely a shared expensive fixture
  • test_inline_extraction::test_a_spawned_pool_is_the_forked_pools_output 18.1 s

Acceptance: before/after full-suite part wall times on a typical ready PR (code-only and docs-only), and the runner-minutes per run, in each PR body. Test count on main and nightly unchanged.

Ngôn ngữ chính
Python
Star
57
Fork
11
Merge trung bình
1 giờ 14 phút
Pull request đã merge (30 ngày)
967

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của squid-protocol/gitgalaxy

Tất cả issue của squid-protocol/gitgalaxy

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.