Default annotation timing gate fails under host CPU contention during deploy
Maintainers usually reply within 1 day
A pull request for this has already been merged.
- #688 by @iskandr — merged
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 70/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- performance, testing
Research direction
Read tests/test_timings.py::test_effect_timing, then compare how test.sh runs it with the standalone CI step “Check default annotation throughput before the full suite.” Inspect test.sh and deploy.sh to trace where xdist runs relative to the timing check. Choose an approach that avoids host-contention false failures while retaining a meaningful budget, and verify the test and deploy flow.
Written by the indexing model from the issue text.
Description
tests/test_timings.py::test_effect_timing measures wall-clock time for 100 random GRCh38 variants against a 10 s budget. Inside ./test.sh it runs alongside the xdist workers, so heavy load from other processes on the machine can fail a release gate even when the code hasn't slowed down.
Observed while deploying 11.3.0 (c580ae4) on macOS ARM with 10 cores and 8 xdist workers, while other processes held the 15-minute load average near 18:
Elapsed: 15.2522 for 100 variants
AssertionError: Should be faster than 100ms / variant!
deploy.sh stopped before upload, as it should. Run alone right afterwards (load ~4), each commit was timed three times:
| commit | elapsed for 100 variants |
|---|---|
11.2.0 3443b6d |
1.19, 1.14, 2.04 s |
11.3.0 c580ae4 |
1.43, 1.25, 1.73 s |
So there was no regression, and the deploy retry passed (4,407 tests).
CI already runs this check as its own step before the full suite ("Check default annotation throughput before the full suite"). Possible fixes:
- run it the same way, before xdist, in
test.sh/deploy.sh, or - time the subprocess's CPU (
time.process_time()) rather than wall-clock. This ignores outside contention but also misses I/O-bound slowdowns.
Either keeps the budget meaningful without retrying deploys by hand.
- Dominant language
- Python
- Stars
- 88
- Forks
- 28
- Avg merge
- 2h 40m
- Merged PRs (30d)
- 50
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from openvax/varcode
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
Maintainers usually reply within 1 day
Similar issues
-
bump patch
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
autowarefoundation/autoware_carla_scenario#1321 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 1-3 hours Newbie friendliness 78/100
QuantConnect/Documentation#2838 ·
Maintainers usually reply within 1 day
-
get_all_subclasses must deduplicate by class identityPossibly taken @Gambit-Checkmate claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 15 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
awslabs/route53-dynamic-dns-with-lambda#67 · 1 comment ·