Split tests into smaller sub-tests
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
No files or tests are named. Start by locating the current one-test-per-function structure and reviewing how Hypothesis handles examples and xfail behavior; the refactor is done when assertions can be reported and xfailed independently without breaking assertions that depend on earlier checks.
Written by the indexing model from the issue text.
Description
Right now there is one test per function. This has worked so far, but it has limitations. The main problem is that if a single thing fails in a test, you can't really get at any of the assertions that come after it. And even if hypothesis could generate separate examples for separate assertions, if you are maintaining a library and you want to xfail a given assertion, you have to xfail the whole test.
This is going to be important for reporting, because it will make it easier to report on exactly which things a given function isn't compliant on. However, this is also a pretty big refactor. It also could be challenging in some cases if some assertion actually depends on a previous assertion working to make sense (e.g., maybe some assertion will implicitly assume a given shape because it was already asserted). For this reason, I would consider this to be lower priority for now, and not worry about implementing it until it becomes needed for reporting (although it certainly doesn't hurt to start thinking about how it might work).
- Dominant language
- Python
- Stars
- 74
- Forks
- 54
- Avg merge
- 4h 35m
- Merged PRs (30d)
- 4
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from data-apis/array-api-tests
-
Difficulty 2/5 1-3 hours Newbie friendliness 87/100
data-apis/array-api-tests#465 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
data-apis/array-api-tests#460 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
data-apis/array-api-tests#455 · 2 comments ·
-
tracking
Difficulty 5/5 Over a week Newbie friendliness 25/100
data-apis/array-api-tests#381 · 9 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 38/100
data-apis/array-api-tests#379 · 10 comments ·
All issues in data-apis/array-api-tests
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
xinnan-tech/xiaozhi-fde-talk#263 ·
-
rules
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
huggingface/Repo2RLEnv#163 · 1 comment ·
-
Difficulty 1/5 Under an hour Newbie friendliness 95/100
huggingface/sentence-transformers#4074 ·
-
comp/dashboard invalid P3
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
NousResearch/hermes-agent#121143 ·