[RFC]: add APIs for array equality to a scalar
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 30/100
Línea de trabajo
Comienza revisando las operaciones de igualdad existentes de la API de arrays y el issue enlazado de Zarr para entender los casos de uso propuestos. Una contribución completa requeriría una API acordada para las comprobaciones de igualdad total, desigualdad total y desigualdad probable, junto con los cambios correspondientes en la especificación y las pruebas.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Today, there are a few cases where people might want to check equality to a scalar.
import numpy as np
is_all_zero = np.all(arr == 0)
However, this code is terribly inefficient for out of memory array.
It can force them to go through the entire memory.
I think that we can do much better to define an operation that will check for
- All equal
- All not equal
and those can be implemented in more "streamed" fashion, that allow the underlying implementation to not create a full boolean array for arr == 0.
Second, I would like to propose an API for "likely not equal", where the strick inequality is not guaranteed.
For data compression, we might just be intrested in learning if the dataset is worth compressing or not:
Zarr for example does this:
https://github.com/zarr-developers/zarr-python/issues/3627
For example, consider the task to compress a 800MB array.
Zarr today:
- Checks for equality to zero
- If all zeros, it skips things
- if not zeros, it compresses things
But if you have an out of memory data array, it may be hard to guarantee that things are not all zero, but the check likely doesn't matter, since with modern compression algorithm all zeros can be efficiently compressed.
>>> import numpy as np
>>> import numcodecs
>>> a = np.zeros((100, 1024, 1024), dtype='float64')
>>> len(numcodecs.blosc.compress(a, b'zstd', 7))
67216
On my computer, the equality check is roughly the same order of magnitude as the compression itself:
In [14]: %time numcodecs.blosc.compress(a, b'zstd', 7);
CPU times: user 2.04 s, sys: 903 μs, total: 2.04 s
Wall time: 266 ms
In [15]: %time np.all(a==0 );
CPU times: user 21.4 ms, sys: 109 ms, total: 130 ms
Wall time: 140 ms
so if zarr misses one of my images, because I think it has non-zero element, its likely not the end of the world in terms of system performance, but if I have to "guarantee that the images are non zero" that can be a costly operation that can't be done through my implementation easily "without checking every single element".
For not, without these APIs, today, zarr does something like:
- Create an empty array
- Check for equality with that empty array
Alternative:
I could likely check numpy metadata like strides and create my own fastpath for this:
>>> zero = np.zeros(1)
>>> a, zero = np.broadcast_arrays(a, zero)
>>> zero.strides
(0, 0, 0)
but this seems more like a hack and would be implicitely redefining "equality operation" to "likely equal" which isn't correct.
- Lenguaje dominante
- Python
- Estrellas
- 281
- Forks
- 52
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de data-apis/array-api
-
Dificultad 1/5 1-3 horas Aptitud para principiantes 88/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
-
Dificultad 5/5 Más de una semana Aptitud para principiantes 35/100
-
Maintenance
-
Dificultad 5/5 Más de una semana Aptitud para principiantes 25/100
Todos los issues de data-apis/array-api
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 88/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100