Pain points in using the test suite

Open
#381 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Quiet
Tech stack
python
Domain
testing

Research direction

Start by reviewing the linked issues 197 and 379, pull request 380, and the references to gh-329 and gh-169. Run the test suite with a reported backend such as JAX or PyTorch to identify a specific ergonomics problem, then confirm the proposed improvement with the maintainers before treating it as done.

Written by the indexing model from the issue text.

Description

tracking

There were reports of several pain points with using the test suite. This is a central tracker for potential improvements of the test suite ergonomics. Things previously reported:

  • difficulty in tracking expected failures across libraries. ISTM this is now as good as it gets, I don't know how to improve on the current --skips-file and --xfails-file usage.
  • https://github.com/data-apis/array-api-tests/issues/197 reported that the test suite is prohibitively slow on jax. Unclear if it is still a problem after the worst offenders were rewritten a while ago. There are several potential ideas in the issue; what's not clear ATM is whether this is still a problem. And if it is, what are the slowest tests and where is the time spent (for instance, for torch I observed that the slowest tests spent majority of time in the internals of hypothesis).
  • https://github.com/data-apis/array-api-tests/issues/379 reported that it is difficult to interpret failures, and https://github.com/data-apis/array-api-tests/pull/380 works towards improving the reporting.
  • gh-329 and gh-169 reported issues with optional extensions

If you're using the test suite, we'd like to hear from you. Please add the pain points and/or suggestions for improvement to this issue.

Dominant language
Python
Stars
74
Forks
54
Avg merge
4d 15h
Merged PRs (30d)
6

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from data-apis/array-api-tests

All issues in data-apis/array-api-tests

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.