Feature request: safety factor parameter

Open
#135 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
python, pytorch
Domain
testing

Research direction

Start by reading the linked Ivy helper at ivy_tests/test_ivy/helpers.py around line 1793 and inspect how boundary values are generated for the remainder case. Clarify the desired safety-factor API and scope with maintainers; done should include documented behavior and coverage showing generated values avoid or intentionally include representable limits.

Written by the indexing model from the issue text.

Description

We (at Ivy) have observed that small variations in how different frameworks handle values very close to the borders of the representable limits of data types can lead to failing tests in the Array API test suite. These errors can be hard to handle, and can result from something as simple as a small rounding error which pushes things over the representable limits for one framework but not another one. For example, with remainder in PyTorch:

# Data generated in `test_remainder`:
x1 = tensor([[ 0.0000e+00, 2.1132e+270,  0.0000e+00,  0.0000e+00],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00]],
       dtype=torch.float64) 
x2 = ivy.array(1.1754944e-38, dtype=ivy.float32)

# Result for PyTorch:
tensor([[0., nan, 0., 0.],
        [0., 0., 0., 0.]], dtype=torch.float64)

i.e. remainder(2.1132e+270, 1.1754944e-38) = nan when it should return 0. We think there has been some rounding of 1.1754944e-38 to 0 which results in the nan as this number is at the 'tiny' limit of float32:

finfo(resolution=1e-06, min=-3.40282e+38, max=3.40282e+38, eps=1.19209e-07, tiny=1.17549e-38, dtype=float32)

Therefore, it would be useful if there was a safety_factor parameter, or something similar, which prevents values from being generated which are right at the limits of the data type. For example, if values are only generated up to 95% of the maximum size (either on a linear or log scale), then this makes it possible the define the importance of such rigorous testing. Such an approach makes it possible Ivy tests to use hypothesis without needing to solve hundreds too many rare corner cases (example).

Let me know what you think 😊

Dominant language
Python
Stars
74
Forks
54
Avg merge
4d 15h
Merged PRs (30d)
6

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from data-apis/array-api-tests

All issues in data-apis/array-api-tests

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.