CI planner: derive shard counts and timeouts from measured durations and the 20-job pool (generalizes #5882)

Open
  • Is something wrong?
#5,919 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
github-actions, python
Domain
ci-cd, devops

Research direction

Start by reading tests/tools/det_sweep_plan.py and ci_layout.py, then inspect the referenced sharded workflow plans and duration caches (#4847/#5048). Define the shared planner around measured shard durations, pool limits, setup cost, and missing measurements; roll it out in the order listed. Done means shard workflows no longer hardcode shard counts or timeouts, and the stated seven-day balance and timeout criteria hold.

Written by the indexing model from the issue text.

Description

ci-cd claim:nyx enhancement priority: medium scope:infra

Same pattern as #5882, generalized, and the CI half of squid-protocol/promoter-bot#34.

Problem

Every sharded workflow fixes its own shard count and timeout-minutes by hand. They are re-measured after something times out or crawls:

  • Det Sweep DB2_MAX_SHARDS=4, with the db2 timeout raised 18 -> 30 (#5869, #5882).
  • CICS Crucible shards [1,2,3,4] (#5530).
  • Doom Rosetta SHARDS: "4".
  • Smoke --splits 3 (#5133).
  • Full Suite Gate per-leg splits / timeout JSON, re-capped four times (#4576 -> #4903 -> #5155 -> #5766 -> #5776; #5156 open).
  • Evidence Refresh Db2 / sweep / records parts.

Only Ground Truth (ci_layout.py, #5616) and the det planner's narrow path size themselves. The pool is fixed (20 concurrent GitHub-hosted jobs) while cases and suites grow every day: Db2 cases went 45 -> 70 in 4 days.

Proposal: one planner, measured inputs

A small shared module (next to tests/tools/det_sweep_plan.py / ci_layout.py) that each sharded workflow's plan job calls:

  • Shards = ceil(estimated work / target shard minutes), capped by the workflow's share of the 20-job pool. The share can shrink under queue pressure, as ci_layout does today.
  • Timeout = p95 of the measured shard duration × margin (e.g. 1.5), floored and ceilinged. It is emitted by the plan, not hardcoded in YAML.
  • Durations: every shard writes measured per-case or per-part seconds into the durations cache (#4847 / #5048). Cases without a measurement fail the plan loudly after a grace period instead of silently unbalancing shards (the 31-of-70 Db2 cases in #5882).
  • Fixed setup cost (e.g. Db2's ~6 min start) is part of the model, so the planner trades shard count against startup.

Order

  1. Db2 in Det Sweep and Evidence Refresh. This is #5882's asks 1, 2 and 4 via the planner.
  2. CICS Crucible and Doom Rosetta.
  3. Smoke parts (#5133) and Full Suite Gate legs (#5156).

Acceptance

  • No sharded PR workflow has a hand-set shard count or timeout-minutes for its shard jobs.
  • A 7-day window with case growth shows no shard timeouts and a slowest/fastest shard ratio ≤ 1.3.
  • Adding cases needs no YAML edit.

Related: #5882, #5869, #5156, #5133, #5135, #5616, #5262.

🤖 Generated with Claude Code

Plan (Opus rescue of PR #6081)

Already merged: #6037, #6059 (Db2 counts, timeouts and measured setup in Det Sweep and Evidence Refresh), #6050, #6068 (CICS Crucible and DOOM-rosetta). Closed: #6081. Its approach was to have det_sweep_plan.py read the live queue itself, and that was wrong.

Steps, in order. Each step is one Sonnet-sized PR of at most 5 hand-written files.

  1. Det Sweep queue depth comes from the workflow, not the planner. This is a workflow edit, so it needs an allow_workflows item or the owner.
    • Add actions: read to the plan job in .github/workflows/det-sweep.yml.
    • On a pull request, both plan steps ("Plan the sweep" and "Plan from the port diff") get the waiting count from python3 tests/tools/ci_layout.py with GH_TOKEN set. Add a small --print-waiting flag there if needed. Pass the count to det_sweep_plan.py with the existing --waiting. An empty value means calm.
    • Do not change det_sweep_plan.py's imports.
    • Acceptance: a test pins the step wiring; test_ground_truth_plan.py and test_det_sweep_fast.py stay green; the plan summary of a real PR run shows waiting.
  2. Det Sweep non-Db2 shard count from durations. Replace the fixed MAX_SHARDS = 6 / PER_SHARD = 3 in det_sweep_plan.py with ci_shard_plan sizing, which det_sweep_plan.py already imports.
    • Acceptance: unit tests at 45, 70 and 140 cases keep the slowest/fastest ratio at or below 1.3 and stay within the pool cap.
  3. Smoke parts (#5133). smoke-test.yml hard-codes --splits 3 and the part job's timeout-minutes: 30. Plan both from .test_durations with a planner module next to ci_shard_plan.py. This is a workflow edit, so it needs allow_workflows.
    • Acceptance: no literal split count or part timeout is left in smoke-test.yml, and a test asserts that.
  4. Full Suite Gate legs (#5156). Replace the per-leg splits / timeout / step_timeout JSON in full-suite-gate.yml with a plan-job output. Per-OS slowdown factors come from the measured leg times. This is a workflow edit, so it needs allow_workflows.
    • Acceptance: no hand-set leg JSON is left.
  5. Calibration check (no code). Once steps 1–4 have run for 7 days, read the shard timings and confirm there are no shard timeouts and the slowest/fastest ratio is at or below 1.3. Post the numbers here. Then close the issue.

Pitfalls the failed attempts hit

  • test_ground_truth_plan.py follows tool module names as plain words through any file the ground-truth tests reach, and test_det_sweep_fast.py reaches det_sweep_plan.py. Naming or importing ci_layout, post_merge_dispatch and the like in det_sweep_plan.py, even in a comment, fails test_no_skippable_tool_is_reached_by_the_jobs_tests. Keep queue and API reads in the workflow.
  • det_sweep_plan.py --event defaults to pull_request. Any live read keyed on that event fires inside unit tests.
  • Evidence Refresh's lane uses a sparse checkout. Anything det_sweep_plan.py imports there must be in that checkout, or be imported lazily with a fallback.
  • Workflow edits need an allow_workflows item or the owner. Two runs here shipped Python changes that do nothing until a later workflow edit. Put the wiring and its test in the same PR.

What to reuse from the closed branch (churn/5919-20261022T2258Z): the test shape in test_ci_shard_plan.py (--waiting wins, an unreadable queue plans calm), moved to test the workflow step instead.

Dominant language
Python
Stars
57
Forks
11
Avg merge
1h 35m
Merged PRs (30d)
975

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from squid-protocol/gitgalaxy

All issues in squid-protocol/gitgalaxy

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.