Providing reproduction code for failed test cases

Comienza con la notificación de errores en torno a array-api-tests y el ejemplo de array_api_tests/test_manipulation_functions.py::test_repeat; rastrea cómo se gestionan actualmente las llamadas a funciones fallidas y las opciones de CLI. Se considera terminado cuando se muestra una línea de reproducción opcional para los errores aplicables y las funciones ausentes utilizadas para configurar las pruebas producen un error claro, con pruebas que cubren ambos comportamientos.

Hi all!

Here's one idea on how to potentially improve interacting with failing tests in array-api-tests suite which @ev-br and I discussed this week.

I've been involved in the introduction of array-api-tests suite in a few repositories and, from my personal experience, the activity that took the most of the time was figuring out root causes of failed tests.

Each failure shows the stack trace (or multiple stack traces, if there were e.g. 3 distinct failures for a given test), but at times they are merely related to the actual root cause. For instance, in NumPy we have an xfail:

# fails on np.repeat(np.array([]), np.array([])) edge test case
array_api_tests/test_manipulation_functions.py::test_repeat

which originally was reported by the test suite with an error:

Cannot cast array data from dtype('float64') to dtype('int64')

I had to manually recreate the function call with the exact inputs to understand which edge case we hit. And for each test the error message was either accurate, like missing keyword argument, or irrelevant - which was mostly related to array scalars, Python builtins, or 0-D arrays as inputs or outputs.

From my point of view, one possible improvement of this process could be something like a CLI option --with-repro-snippets, where each failing test is accompanied by a copy-paste line that exactly reproduces it.

So for test_repeat I would get:

Here's a line that reproduces it:
xp.repeat(xp.asarray([]), xp.asarray([]))

The array-api-tests suite would compose it with f"xp.{func_name}({inputs}, {kwargs})" when a function call fails.

This way, when I run a test suite for the first time and get e.g. 50 failures, I can iterate a bit faster by skipping the "reproduce failing code snippet" step. WDYT? Please share your thoughts!

More of a nitpick but some Array API functions are used in multiple tests, like reshape. I think it’s used primarily for setting up inputs for tests. When that one function is missing from the namespace, a large part of the test suite fails with more cryptic error messages. I think that there could be a hasattr(xp, "reshape") decorator for tests and if a function for setting inputs is missing, the error says so.