Tests are too slow for libraries with eager-mode dispatch overhead
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- performance, testing
Research direction
Start with the linked GitHub Actions job and the array-api-tests suite to reproduce the reported 4.5-hour run and inspect which tests dominate runtime. The issue does not name source files or test entry points; done means substantially reducing eager-mode test time while preserving the suite's compliance coverage and results.
Written by the indexing model from the issue text.
Description
For example this job: https://github.com/google/jax/actions/runs/6804780902/job/18502996983, which is associated with https://github.com/google/jax/pull/16099 took 4.5 hours to run about 1000 tests.
========== 567 failed, 439 passed, 116 skipped in 16419.47s (4:33:39) ==========
This is slow enough to be virtually unusable.
Is there anything I can do to speed up the tests during development of the JAX array API?
Edit: as mentioned below, this is due to the frequent use of patterns within the tests that every value in each array via indexing, and the fact that in eager execution, each of these indexing operations have a small dispatch overhead that leads to slow tests when arrays are large.
- Dominant language
- Python
- Stars
- 74
- Forks
- 54
- Avg merge
- 4h 35m
- Merged PRs (30d)
- 4
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from data-apis/array-api-tests
-
Difficulty 2/5 1-3 hours Newbie friendliness 87/100
data-apis/array-api-tests#465 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
data-apis/array-api-tests#460 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
data-apis/array-api-tests#455 · 2 comments ·
-
tracking
Difficulty 5/5 Over a week Newbie friendliness 25/100
data-apis/array-api-tests#381 · 9 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 38/100
data-apis/array-api-tests#379 · 10 comments ·
All issues in data-apis/array-api-tests
Similar issues
-
bug confirmed issue
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
open-webui/open-webui#30750 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
OpenwaterHealth/openmotion-bloodflow-app#604 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
good first issue
Difficulty 1/5 Under an hour Newbie friendliness 90/100