[DX] choose optimal job size to reach best speed
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- php
- Domain
- performance
Research direction
Start by tracing where the default jobSize and timeoutSeconds values are defined and how worker chunks are scheduled. Reproduce the reported benchmarks for small and large file sets, including the timeout case with heavier rules. Done means choosing and documenting defaults that improve throughput without causing the reported child-process timeouts.
Written by the indexing model from the issue text.
Description
13 111 files
jobSize |
wall |
|---|---|
| 16 | 218 s |
| 150 | 86 s |
| 300 | 75 s |
279 files (a single module of the same project)
jobSize |
chunks | workers | wall |
|---|---|---|---|
| 16 | 18 | 14 | 14.1 s |
| 50 | 6 | 6 | 8.9 s |
| 100 | 3 | 3 | 9.5 s |
| 150 | 2 | 2 | 10.9 s |
| 300 | 1 | 1 | 12.2 s |
On 279 files even a single worker - jobSize: 300, no parallelism at all - beats the current default running on 14, because starting 14 processes costs more than the parallelism returns.
The optimum does move with corpus size (~50 for 279 files, 300+ for 13 111), but 100 is close to both, and anything in 100-200 beats 16 everywhere I measured.
So yes, increasing default job size to 100 might be a good idea, but:
jobSize and timeoutSeconds interact. A worker has to finish a whole chunk inside the per-job timeout, and 16 -> 150 multiplies the work per chunk by ~10. With a heavy rule set and a cold cache I hit exactly that - Child process timed out after 120 seconds, exit 1 - at jobSize: 150, where 16 completed. If the default goes up, the default timeout likely needs to go up with it, otherwise the failure mode moves from "slow" to "red build".
Originally posted by @webard in https://github.com/rectorphp/rector-src/issues/8489#issuecomment-5662569618
- Dominant language
- PHP
- Stars
- 137
- Forks
- 451
- Avg merge
- 7h 11m
- Merged PRs (30d)
- 129
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from rectorphp/rector-src
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
rectorphp/rector-src#8493 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
rectorphp/rector-src#8488 · 1 comment ·
All issues in rectorphp/rector-src
Similar issues
-
tooling
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
UX
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
ProfessionalWiki/NeoWiki#1525 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
OpenConext/OpenConext-engineblock#2122 ·
-
Bug
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Automattic/safe-publish#594 ·